Performance Quiz #1 (of a series?)
Recently there was a discussion on one of our internal email aliases in which this problem came up. I though it was an interesting problem so I posed this Quiz to assorted people I work with to see what kinds of things they would say.
Considering these three options:
Option 1:
sw.WriteLine(subject + ": " + message);
Option 2:
sw.WriteLine("{0}: {1}", subject, message);
Option 3:
sw.Write(subject);
sw.Write(": ");
sw.WriteLine(message);
Answer these questions:
Q1. Which of these choices offers the best performance?
Q2: Can you characterize the memory usage of each of these?
Q3: Which would the performance team generally recommend, and why?
Q4: What special factors might alter this recommendation?
Q5: What did you have do assume about "sw" to answer the question?
I encourage you to think about this for a few minutes (more is good too) before reading past the line. SPOILERS/ANSWERS follow.
For everyone that took the time to think about this at all, thank you very much.
In the course of my job I'm often asked to comment of the probable performance of an assortment of solutions to give guidance, or at least suggest what measurements should be made. So I approached this quiz the same way and made my best guesses/recomendations as I would if I could not do the measurements. Then I went back and did the actual measurements.
Here are my own answers:
Q1. Which of these choices offers the best performance?
- Only thing I can say for sure is that #2 will lose to #1 in all cases
- #3 is going to be best if the output is buffered
- #1 is going to be best if the output is unbuffered
- In any case, a typical program's overall performance will be unaffected by the choice
Q2: Can you characterize the memory usage of each of these?
(These answers all proved to be "close but no cigar" due to the unusual WriteLine behavior discussed in the real analysis, see below)
All three probably have allocations associated with buffering the stream, ignoring those as invariant, the allocations unique to each choice are:
#1 single concat operation, one temporary string
#2 assorted allocations, including string builder, underlying string buffer, vararg array (I was close)
#3 no allocations
Q3: Which would the performance team generally recommend, and why?
Even though it's the worst performing, and we knew that much in advance, both of your CLR Performance Architects concur that #2 should be the default choice. In the highly unlikely event that it becomes a perf problem the issue is readily addressable with only modest local changes. Normally you're just cashing-in on some nice maintainability. We do not alter our choice given the data below.
Q4: What special factors might alter this recommendation?
Specific measurements indicating that the code path had become a hotspot.
Q5: What did you have do assume about "sw" to answer the question?
Only that the stream did not have exotic behavior (such as weird cryptographic features that make the cost model very complex) and that it was buffered. In the event of an unbuffered stream of one type or another there are signficant semantic differences between (1 or 2) and (3) and potentially huge perf differences too.
OK, time to see how we did.
To do the analysis below I used the following benchmark program and CLR Profiler, which is one way to look at this data.
| namespace Test |
| { |
| using System; |
| using System.IO; |
| class Test |
| { |
| static private String s1 = "Hello"; |
| static private String s2 = "Good bye"; |
| static private int iterations = 5; |
| static private volatile int foo = 0; |
| static private MemoryStream ms = new MemoryStream(100000); |
| static private StreamWriter sw = new StreamWriter(ms); |
| public static void Main(string[] args) |
| { |
| int i; |
| for (i=0;i<iterations;i++) Test1(); |
| for (i=0;i<iterations;i++) Test2(); |
| for (i=0;i<iterations;i++) Test3(); |
| } |
| public static void Test1() |
| { |
| sw.WriteLine(s1+": "+s2); |
| foo++; |
| } |
| public static void Test2() |
| { |
| sw.WriteLine("{0}: {1}", s1,s2); |
| foo++; |
| } |
| public static void Test3() |
| { |
| sw.Write(s1); |
| sw.Write(": "); |
| sw.WriteLine(s2); |
| foo++; |
| } |
| } |
| } |
The results below are in the form of an execution trace showing functions and allocations for each of the three options as reported by CLRProfiler.
|
| |||||||||
| |
|||||||||
| |
|||||||||
This is already too long but I thought I'd end by sharing a quick summary of the nature of the responses I got internally.
- Some thought the problem was very easy. Many of these people didn't do very well :)
- Everyone (including me) was suprised by the allocation in #3.
- Some people thought all the options would go through the formatting logic, looking for { and } and so forth, that's not the case if there is only one argument to Write/WriteLine.
- Some people feared the three function calls they could see in Option 3, but didn't think about the internal work assocated with formatting in #2 that they could not see. There was a great tendancy to assume WriteLine was magic or rocketscience -- generally a bad idea. The three virtual function calls are really the least of your problems with #3.
- Some people assumed no buffering (fair enough), which majorly colors the answer.
- Some had supreme faith that we had done magic in #2 to make it perfect... alas no.
- And lastly, Some were afraid to guess at all, must be people that read my blog :)
Comments
- Anonymous
March 12, 2004
What a great post! I love this kind of information. I also find it interesting that even at Microsoft developers make the same kind of assumptions the rest of us make. You should defently make a series out of this. This is also the type of column/information that msdn.microsoft.com should carry. Thanks! - Anonymous
March 12, 2004
I would also be curious to see the numbers if yu had used a StringBuilder to construct the string and then write it out. - Anonymous
March 12, 2004
Actually if you look at the trace for the WriteLine with formatting carefully (sorry about that small font) you'll see that what it did was precisely what you suggest -- it internally used a StringBuilder.
In fact I'm pretty sure that the only thing that actually has general purpose formatting services in its implementation is StringBuilder so everything else that offers general purpose formatting is in fact using a StringBuilder to get the job done. - Anonymous
March 12, 2004
An added bonus for #2, it is a lot easier to localize since #1 and #3 assume an specific string ordering.
Great post. Even though I expected #3 to be the fastest, I would not have predicted the difference in function-calls/bytes between #1 and #2. - Anonymous
March 16, 2004
Tom's log :: .NET string performance - Anonymous
March 16, 2004
good article. more like this please! - Anonymous
March 16, 2004
Oopsy.
These were my quick answers before reading the spoilers.
Q1 : Best Perf #1 But # 3 is more correct.
Q2 : Memory , #2 uses more
Q3 : #3, it will remain similar as you add more and more writes.
Q4 : The size of the strings, the size of the string builder and how much buffer it has waiting.
Q5 : I am assuming it is a StringBuilder. - Anonymous
March 16, 2004
[.NET]Performance Quiz #1 (of a series?) - Anonymous
March 17, 2004
Can you really safely draw conclusions about "best performance" without measuring CPU usage? I know calls and bytes allocated is a good indicator of performance, but I would have expected something like "how long does it take to do it ten million times" to be a more accurate test. How do you decide when a test is accurate enough for its results to be actionable? - Anonymous
March 17, 2004
That's an excellent question, but rather than just spout a few quick words here. I think I'd like to write a seperate blog about making and interpreting benchmarks. I'll see if I can give you something substantial soonish. - Anonymous
March 18, 2004
The comment has been removed - Anonymous
March 19, 2004
Interesting, but it seems to raise more questions....
Your test uses concatenation of three strings -- which is apparently treated as a special case by both the C# compiler and the CLR. What happens when one tries concatenating 4 5 or 6 strings?
On the other hand, what would appear to be a special case optimazation with String.Format (using direct parameters rather than an array) turns out to be merely syntatic sugar covering a DE-optimization (building an array behind the scenes). So, why are you optimizing the method you tell us not to use (adding strings), instead of the method you tell us TO use (StringBuilder.Format)? Is there a point (number of parameters) where String.Format beats Concats? And, if the non-array versions of String.Format are just wrappers around the array version, why limit it to just three parameters? Why not 4, 5, 6 etc parameter versions? - Anonymous
March 22, 2004
The comment has been removed - Anonymous
March 23, 2004
The comment has been removed - Anonymous
March 29, 2004
Good.
But I think writeline("{0}:{1}", str1, str2) should be used when you might need to change the format of the output. For example, maybe one user would like the output {0}:{1}, but some others may like the output to be [#{1}]:{0}.
In a word, writeline(str1 + ":" + str2) is so called a hardcoding, which maybe fast in coding but expansive in maintaining. - Anonymous
May 17, 2004
Question:
I carried out the same profiling for the same code with broadly the same results. However, the first iteration of Test1 allocated 586 bytes, the following 4 allocated 94. The first iteration of Test2 allocated 428 bytes, the following 4 allocated 184. Why does the first iteration allocate more memory? Is the CLR being clever and reusing objects allocated on the first iteration? - Anonymous
May 18, 2004
Nothing exotic, I believe what's going on there is that there are certain string literals that get allocated the first time through the code and can subsequently be re-used.
There may be other static class members that need initialization the first time through in the same way.