Share via

How not to benchmark different languages

  • Article

I've recently been trolling the web for any sort of language-comparision benchmarks, to see how the CLR's JIT stacks up to the competition.  Dr. Dobbs has what seemed to be a pretty reasonable micro-benchmark article.  It's not particularly insightful, but hey, it is hard to come up with stuff that makes sense to compare across multiple disparate languages.  I started looking for specific C# to Java comparisons and came upon this little gem.  I'll give you a few minutes to read it.

Now that you've read it, I would like to challenge you to understand how anyone would ever believe the question of "How long does it take to do nothing 1 billion times" in any language is worth comparing.  Seriously, ignore the thoughts regarding whether the comparison itself was fair or not.  The best part of that article is that the Java JIT they seem to be pushing doesn't get rid of the empty loops in a newer version.  Then look at their 'testing methodology' (or the lack thereof).  Not only are they not actually benchmarking anything that anyone with a brain in their head would care about, but they didn't even bother to figure out how use the tools that they claim to be comparing against.  Does anyone that actually uses C++* think that if they care about performance they're not going to enable the optimizer?  Every reasonable C++ compiler on the planet will completely eliminate the loops if only told to optimize.  And then the comparison against Assembly stuck in my craw.  Assembly is there because you KNOW WHAT YOU'RE DOING!!!!  If an assembly programmer writes a loop that iterates for a billion times, it's probably because they're trying to measure some particular pattern of code on the CPU they're running.

So (still snickering at the previous stuff), I guess I'm asking for input:  Does anyone have any non-micro-benchmark relatively objective managed code comparison links they'd recommend?  I'd really like to see how the CLR's JIT stacks up against some of the JVM's out there, for more than just little tiny toy code samples.

I'm also interested in 'end-to-end' timing, which includes JIT time.  Microbenchmarks with internal clocks don't tell you how long something actually ran for.  If a JIT takes 5 minutes to figure out it can remove 2 dead loops, your runtime isn't less than 1 second - it's less than 5 minutes and 1 second.

-Kev

* I found 4 different syntax errors in the C++ source code they provided...

Comments

  • Anonymous
    July 16, 2007
    The biggest issue by far wrt. CLR performance is this: http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=93858 I am totally amazed why nobody bothers to fix this issue. It could literally make a factor of 5 difference with some very common code such as handling System.Drawing.PointF or Microsoft.DirectX.Vector3. I mailed Rico Mariani several times about this issue, and he agrees with me. But it seems that nobody at MS cares about performance anymore.

  • Anonymous
    July 16, 2007
    I'm happy to say that we are currently in the process of fixing the inlining limitation in the 32 bit JIT.  The 64 bit JIT shouldn't be too far off, either.  When the new JIT sees the light of day is anyone's guess [it's too late for Orcas, and likely won't be in Silverlight, either :-(]  But, hey, we have actually fixed this limitation!

  • Anonymous
    July 17, 2007
    Check out the language shootout: http://shootout.alioth.debian.org/ Lots of microbenchmarks, and a wide variety of languages. The .NET CLR is not benchmarked because it is not available for linux though. Mono does OK, but not great.

  • Anonymous
    July 26, 2007
    I've waited 6 years for the inlining limitation to get fixed, I can hold out for a couple more :) Are there any plans for blog entries regarding JIT?

  • Anonymous
    October 15, 2007
    Any update on when this fix will be released? I did not have any luck finding an up to date roadmap for the CLR anywhere on the web.

  • Anonymous
    October 16, 2007
    <caveat level="heavy"> The new inlining heuristics are done and checked in to a main source tree.  They are guaranteed to be in the next major release of the CLR.  As release dates & details are not yet public, I probably can't say much here.  Sorry!  When they become public, I'd be happy to mention it in my blog. </caveat>

  • Anonymous
    November 28, 2007
    I understand that you can not tell us confidental information about release schedules. But why is it that a huge company like microsoft does not have a public roadmap for one of their core technologies? All the latest .NET framework releases added more class libraries, but did not do any changes to the CLR. I personally could not care less about stuff like workflow foundation or whatever. I just want a fast and efficient core infrastructure. Is that too much to ask for?

  • Anonymous
    January 16, 2008
    You could have emailed me, Kevin. Here's an example, albeit old...http://www.shudo.net/jit/perf/ Other examples available upon request. I'll tell you over coffee what the top 5 issues with the CLR JIT are...if you dare.