Performance: What Common Benchmarks Measure

Many enthusiasts want a fast browser, and they want to consistently and easily measure the performance of different browsers. There are lots of “benchmarks” in the history of software, and lots of benchmarks around browsers. The challenge with benchmarks is generally understanding what the benchmark measures and deciding whether those measurements are important to customers, developers, and you.

This blog post takes a closer look at some of today’s most common benchmarks. We’ll provide context on their origins, talk about what they test, and show you what browser subsystems each benchmark executes.

Celtic KaneCeltic Kane is generally credited with being the first JavaScript benchmark. It was originally created in 2006 by a web developer (Sean Patrick Kane) and became popularized after the benchmark was featured on Digg.com. Celtic Kane consists of eight simple test cases which are each run once and measured in milliseconds. Celtic Kane claims to focus on core JavaScript and DOM performance; however the DOM tests inside Celtic Kane don’t actually measure the DOM.

WebKit SunSpider
WebKit SunSpider is one of the most commonly cited benchmarks and has become a focus for the industry. Given that SunSpider focuses on JavaScript performance and not real world scenarios that matter to customers this is unfortunate. SunSpider is developed by the Apple WebKit team with the goal of measuring real world JavaScript performance. There are 26 test cases which attempt to measure the performance of individual JavaScript features including arrays, math, and regular expressions. The WebKit SunSpider tests exercise less than 10% of the API’s available from JavaScript and many of the tests loop through the same code thousands of times. This approach is not representative of real world scenarios and favors some JavaScript engine architectures over others.

Google V8
Google’s V8 was designed to help Google tune the V8 JavaScript engine. The benchmark carefully targets the JavaScript subsystem and measures some computationally complex scenarios such as OS thread scheduling, cryptography, and ray tracing in JavaScript. Though well designed, some of the V8 benchmark tests were programmatically generated from Scheme language benchmarks and do not use programming language patterns commonly used by JavaScript developers. While academically interesting these scenarios are not representative of how the JavaScript language is used or fairly represent patterns across the web.

Mozilla Dromaeo
Mozilla’s Dromaeo is one of the best engineered JavaScript and DOM benchmarks available today. It was originally written by John Resig and the Mozilla Foundation to support Firefox development and is a very effective tool for browser vendors who need to measure the performance of individual API’s. In addition to pure JavaScript and DOM tests, Dromaeo measures a few additional areas including common jQuery and Prototype features along with CSS Selectors. Dromaeo also provides a test harness for the WebKit SunSpider and Google V8 benchmarks.

Flying Images
The Flying Images demo released with the first Internet Explorer Platform Preview has unexpectedly become a “hardware accelerated graphics” benchmark across the web. Our intent with Flying Images was to show how the GPU can impact the HTML4 websites of today, not create a benchmark. That said it does fairly represent the underlying coding patterns found in today’s common AJAX applications. Flying Images uses standard HTML4, JavaScript, and CSS to programmatically move images around on the screen every 1/60 of a second (considered real-time on a 60hz display). It manipulates the DOM, changes CSS properties, and causes the entire page to go through the layout process.

FishIE Tank
The FishIE Tank demo released with the third Internet Explorer Platform Preview was treated by some bloggers and technology reviewers as a benchmark because of the visually compelling scenario. FishIE Tank merges both traditional HTML coding patterns with HTML5 Canvas. It uses HTML5 Canvas to animate fish around the screen with a JavaScript based collision detection system, and HTML elements and common HTML coding patterns such as setting innerHTML to display an FPS meter over the tank. This demo is representative of the coding patterns that we’ll see as developers build applications that merge the established HTML and emerging HTML5 constructs.

Psychedelic Browsing
The Psychedelic Browsing demo has received over 500,000 page views over the last five weeks since the fourth Internet Explorer Platform Preview was released in August. Psychedelic Browsing spins a color wheel as quickly as possible on the screen and was designed to stress a common HTML5 coding pattern that uses JavaScript and the DOM to draw directly into an HTML5 Canvas while playing an HTML5 Audio element in the background. We have seen this pattern start to quickly emerge over the last six months as Canvas has seen broad adoption. This is representative of where many web applications are headed.

We believe that to build a fast real world browser you have to start by understanding the real world patterns and design the right systems around those patterns. You can’t build a fast real world browser by optimizing for benchmarks. And you can’t accurately assess a browsers performance by looking solely at the results from benchmarks. When used for engineering purposes, benchmarks can be an effective tool to help understand the implications of code changes. That’s where the value ends though.

As an industry we have to care be careful that we don’t lose sight of what’s important – the real world customer scenarios of today and the HTML5 scenarios of tomorrow.

Jason Weber
Lead Program Manager, Internet Explorer Performance

Comments

• Anonymous
  September 14, 2010
  Just one more day till IE 9 beta! This will shame mozilla into finishing JM eventualy.

• Anonymous
  September 14, 2010
  The comment has been removed

• Anonymous
  September 14, 2010
  IE9 (platform preview 4) does perform very well against the above noted demos - but it is important to note that shipping versions of IE (and the last available version of IE for XP) outright FAIL the latter tests, and come in dead last on the first tests. Although that speaks volumes for the advancements that IE has made in IE9 and latching on to the GPU to overcome it's performance issues be careful how much you brag about this as your current user base is using a FAIL-ing version of your browser (and a vast quantity of them can never upgrade their IE version - ever)

• Anonymous
  September 14, 2010
  While IE9 is doing very well on these tests that test more subsystems and the implication is that IE9 is overall faster than other browsers that have faster javascript engines, in real world use there are currently virtually no web sites like this. The real test of a browser is for pages that people use:

• How fast it loads
• How snappy it feels A benchmark that tests this would be a better indicator of browser performance as it really matters to users (kraken perhaps?). It has been mentioned that as IE is not cross platform there are various speed advantages, but until it can claim to be faster than other (beta) browsers in current real world use, there doesn't seem to be much room for making claims of speed. (In 3+ years when "HTML5" is prolific then these tests may be more meaningful)

• Anonymous
  September 14, 2010
  While IE9 is doing very well on these tests that test more subsystems and the implication is that IE9 is overall faster than other browsers that have faster javascript engines, in real world use there are currently virtually no web sites like this. The real test of a browser is for pages that people use:

• How fast it loads
• How snappy it feels A benchmark that tests this would be a better indicator of browser performance as it really matters to users (kraken perhaps?). It has been mentioned that as IE is not cross platform there are various speed advantages, but until it can claim to be faster than other (beta) browsers in current real world use, there doesn't seem to be much room for making claims of speed. (In 3+ years when "HTML5" is prolific then these tests may be more meaningful)

• Anonymous
  September 14, 2010
  You missed the PeaceKeeper Benchmark service.futuremark.com/.../run.action

• Anonymous
  September 14, 2010
  >(kraken perhaps?). roflmao... yeah, running FFTs on audio and images is really what "real-world" websites do.

• Anonymous
  September 14, 2010
  Good post. Measuring (anything) is a science in itself, and I'm sure that you are only scratching the surface here. Just thinking of what you are exactly measuring, and what you actually measure can be a pain. Weighing the measurements of the subsystems. Graphics vs. DOM, for example. Then there's statistics analysis of your data. And then you have to formulate a result. Unfortunately the uninformed internet public can just click any benchmark button, and the one that matches their purpose/preconceived ideas best would be the best and most reliable one. And obviously a score of 42 trumps an 18. That's an uphill battle you can't win

• Anonymous
  September 14, 2010
  The comment has been removed

• Anonymous
  September 14, 2010
  The comment has been removed

• Anonymous
  September 14, 2010
  The comment has been removed

• Anonymous
  September 14, 2010
  There are quite funny comments about "these test does not matter because they are wrong" but for a couple of years those same tests has been very correct to show the superiority of firefox and chrome. Now when IE takes the lead the tests become not valid anymore :D. And what comes to the new tests that takes advantage of GPU, well we are not there yet but at least we can see that IE9 platform preview has way better hardware acceleration than firefox 4 beta 5.

• Anonymous
  September 14, 2010
  Some early results from Kraken... IE9 PP4 http://tinyurl.com/ie9p4kraken Firefox 3.6 http://tinyurl.com/ff36kraken Chrome 6 http://tinyurl.com/cr6kraken Not a great result for IE, lets hope the beta can do better.

• Anonymous
  September 14, 2010
  The comment has been removed

• Anonymous
  September 14, 2010
  The comment has been removed

• Anonymous
  September 14, 2010
  @thenonhacker Can you name one benchmark of performance or standards compliance that IE9 beats Chrome at that isn't created by Microsoft? Peacekeeper (Chrome6: 9433, IE9pp4: 3110 - my pc) Kraken Dromaeo Sunspider V8 benchmark html5test (Chrome6: 217, IE9pp4 97) acid3

• The driving forces behind the web is business and business wants to reach as many people as possible. Until a significant percentage of people have browsers and computers that can handle the IE9 tests, it doesn't make business sense to develop them. This is a long way off and also, no browser will die while there is enough people using it to make it necessary to cater for it. If this wasn't the case IE6 (and 7) would have been dead long ago and it they have been far surpassed by all other browsers.
• The web is primarily about information and communication. It is questionable that there is much use for intensive graphics on a website as movement is going to distract the user from reading the information. If IE9 was the only browser in the world there won't be an explosion of intensive graphics on websites because having them doesn't make sense in at least 90% of cases.
• Quake live provides fully cross browser network gaming via a plugin. Games can be made fully cross browser/platform now without needing a IE9 style browser.

• Anonymous
  September 14, 2010
  "There will be a LOT of sour graping once there will be Game Sites that provide DirectX-quality HTML5 Canvas games." do you really believe that?

• Anonymous
  September 15, 2010
  The comment has been removed

• Anonymous
  September 15, 2010
  @All: Read the article above again. Don't miss his point that a browser should not be just confined to JavaScript Performance. There are more areas than JavaScript that has to be fast. Thanks!

• Anonymous
  September 15, 2010
  @thenonhacker: look at the tests and you will notice that even though the IE tests include DOM and others, the real advantage comes from the hardware acceleration. That will only benefit heavy animation. That doesn't mean you will benefit on other areas as well. The better comparison right now would be load the list of the 100 most visited sites and comparing timeload of them, without cache.

• Anonymous
  September 16, 2010
  When viewing youtube videos in html5 mode, there are significant frame drops at the beginning, it's just really slow (unwatchable). But after that, everything is super-smooth. The first few second uses an interpreted js and after that jitted one is used, correct? Whatever the cause, it's unpleasant. It would be ok if it would happen once (for youtube) but this happens for every new movie

• Anonymous
  September 16, 2010
  The comment has been removed

• Anonymous
  September 16, 2010
  Yah. Despite all the new speed, the IE9 beta doesn't even feel as fluid as my Firefox 3.6.10...

• Anonymous
  September 17, 2010
  @Asrail, Kraken is just pure JavaScript test, there's no DOM, rendering, graphics, etc. etc. test in Kraken, so I don't think Kraken results will be representative of real world applications. I think Peacekeeper is still a far better overall benchmark than Kraken.

• Anonymous
  September 17, 2010
  The comment has been removed

• Anonymous
  September 17, 2010
  @wetran Here's something interesting about the Peacekeeper benchmark - weblogs.mozillazine.org/.../020664.html

• Anonymous
  September 18, 2010
  @NoPeace, Yes the Peacekeeper benchmark does have its quirks and weirdness, but it's still arguably the most thorough benchmark that tests various browser subsystems and approximates real world applications best, since unlike Kraken and Sunspider etc. which only shows some text and blocks, Peacekeeper actually renders real pages and shows real graphics drawing. And unlike those Microsoft testdrive demos, it's not trying to exploit specific bugs (or features?) of a specific browser (namely Chrome), ie. the oversampling problem when downscaling images (which is mostly irrelevant to GPU-acceleration as shown by results from Opera). At least it's an independent benchmark that's not constantly sending out absurdly large images for the browser to downscale at the client-side to exploit a specific weakness in a competitor browser (and web design common sense tells you should downscale the images to reasonable sizes at server-side to save bandwidth and loading time).

• Anonymous
  September 18, 2010
  Just a question for the IE team, does using <meta http-equiv="X-UA-Compatible" content="IE=IE7" /> make the webpage rendering slower than if it was using <meta http-equiv="X-UA-Compatible" content="IE=IE9" /> ? Is the IE7 mode optimised in IE9 to use GPU rendering etc. and make it faster, or does it go back to the speed of IE7?

• Anonymous
  September 21, 2010
  Anyone?

• Anonymous
  September 21, 2010
  @Andrew: Generally speaking, IE9 standards mode is our "fastest" mode. However, we still use GPU rendering, JIT-compiled JavaScript, etc, for IE7-compatibility mode pages.

• Anonymous
  September 22, 2010
  OK, thanks - I didn't realise you replied right after!