Partilhar via

Summarizing Common Browser Tests

  • Artigo

To deliver on the industry promise of “same markup,” browsers must implement features in the same way. Open Web standards and their associated public test suites provide the means for making sure this happens. Using the same process for tests as for the standards themselves is so important that Microsoft contributes to this industry effort in many ways, such as submitting conformance tests to the W3C.

In addition, we believe it’s important for these tests to be comprehensive and test the depth and quality of standards support, not just the existence of features. Compliance tests from standards bodies do that.

Standards Compliance

We’ve written about these tests before. They come from the W3C working groups that write the specifications. The purpose of the test suite is to ensure the specification is complete and has two conforming implementations, the requirement for a W3C specification to become a Recommendation.

Examples of W3C conformance test suites:

The HTML5 Test Suite is particularly noteworthy given the excitement around HTML5 and we’ll blog more about that next week.

Many different types of tests exist beyond those created by the W3C and take various approaches. You’ve probably seen many of these already. While not an exhaustive list, here are a few examples with descriptions of what they test and how.

Feature Existence

Some tests check for just the existence of features. Examples include The HTML5 Test, “When can I use…,” and findmebyIP. They perform a basic feature check, but don’t test the depth of an implementation or how well the feature’s behavior matches the standard. Thee tests rarely reflect the current status of a specification. For example, in the list above, only “When can I use…” lets you filter the view to only features with a chosen W3C specification status.

Some tests in this category compute a total score, though there is no consistent approach to calculating one. While a single score is easy to communicate, it doesn’t help people understand what the score means. When reading these tests, it’s important to see which features are and are not included and to what depth, rather than depending only on a final score.

Depth Testing

These tests attempt to verify more than whether a feature exists, and typically focus on a single specification. They’re usually developed by various people outside the W3C. They can be useful when the W3C working group hasn’t finished their test suite or as additional testing, but there is no assurance they represent the entire specification or are otherwise complete.

Examples:

Compatibility Charts

These suites use test cases to create compatibility information so Web developers can understand the behavior of certain features across different browsers. Similar to the Feature Existence tests, they check support for a feature but typically test the implementation’s correctness to some degree. This is helpful when you need information on a specific element, property, or other feature. Like the previous group of tests, these don’t benefit from the working group’s review and evaluation, and it isn’t clear what they actually test.

Examples:

Other Often-cited Tests

Some often-cited tests are a mix of the kinds described above.

Acid3, for example, is unique in that it tests about 100 fragments of a dozen different technologies. Some parts check for the existence of a feature, some check for specific behavior of a feature, and others check for behaviors where browsers were known to differ when the test was created. Acid3 gives each test one point and computes a final score based on the number of tests passed. While people often communicate the score, it’s unclear what that score represents given the variety of tests included.

Browserscope is a collection of tests in many areas, ranging from security and network behavior to RichText and Selectors API. Many of the tests check if a feature exists, as in the tests for postMessage and JSON.parse. Other tests check for specific behavior. For example, the RichText section tests many small behaviors in different categories. Various groups participate in creating these tests, but there’s no assurance they represent an entire specification and they don’t benefit from a working group’s evaluation.

An Industry Goal

Allowing developers to have the same HTML, CSS, and JavaScript work the same way across many browsers is an industry goal that benefits the developers who build the Web and the people who browse it. Testing is a critical step towards achieving this goal, and that’s why we work closely with standards bodies to create comprehensive test suites so people can assess not just whether a browser supports a standard, but the quality and depth of that support.

—John Hrvatin, Lead Program Manager, Internet Explorer

Comments

  • Anonymous
    February 09, 2011
    It's almost like some of the other guys are taking HTML5 pass/fail...

  • Anonymous
    February 09, 2011
    The comment has been removed

  • Anonymous
    February 09, 2011
    The comment has been removed

  • Anonymous
    February 09, 2011
    "Allowing developers to have the same HTML, CSS, and JavaScript work the same way across many browsers is an industry goal" If I recall, that was the goal when the Web was first conceived.  Write once, run anywhere - remember?  Microsoft - and Netscape but to a lesser degree - are responsible for the HTML fragmentation by using proprietary tags and language when it suited them.  Now you're trying to fix it, fine, but don't make it sound like you're reshaping the world - all you're doing is trying to close the stable door while trying to push the horse back in the barn at the same time.

  • Anonymous
    February 09, 2011
    Microsoft doesn't believe in fixing rendering issues with patches, only security. I find this extremely daunting. We have to wait a whole year or more for another IE and the fragmentation of browsers continues yet again. Firefox and Chrome teams both encourage updating IMMEDIATELY and they support developers creating extensions to ensure compatibility.

  • Anonymous
    February 09, 2011
    @Craig Video rendering is actually IE9's strong point. Part of the reason Chrome removed h.264 support is that they don't want to craft a h.264 decoder that can compete with IE9, since it is much faster, being able to play video more smoothly and using less system resources than chrome could. @Neil Dunensach That wasn't the goal at all. If it was, html and technologies would have stagnated. Html was established to be a markup language which could easily to extended and interact with new technologies. The reason why html was able to supplant gopher as the definitive protocol of the internet was because the Mosaic browser introduced a then proprietary element, the <img> tag which made that browser and html the killer app of the time. Without fragmentation, HTML would be dead.

  • Anonymous
    February 09, 2011
    Andrew, Near each month Microsoft releases a cumulative patch to IE and it usually doesn't only fix security issues, it always comes with several non-security related fixes: you can check their descriptions in the unique KB article of each cumulative patch. But I agree that Microsoft should make this information more accessible to the public that keeps believing that Microsoft only fixes security issues between each IE first number version change.

  • Anonymous
    February 09, 2011
    The comment has been removed

  • Anonymous
    February 09, 2011
    The comment has been removed

  • Anonymous
    February 09, 2011
    Google does it again... code.google.com/.../Welcome trips Compatibility View in IE9 Beta...(see Doctype link above in your post). Appears that they (Big G) don't bother to test in IE (any version) <meta content=IE=edge,chrome=1 http-equiv=X-UA-Compatible>

  • Anonymous
    February 09, 2011
    You forgot the ecmascript test suite http://test262.ecmascript.org/ Cheers,

  • Anonymous
    February 09, 2011
    @Anonymous - IE9's HTML5 video will be the defining point of the downfall of the Open Web.  Microsoft will be squarely to blame and developers will find a new level of "low" respect for IE. The RC is coming out today - I seriously hope that Microsoft is seriously re-considering just how foolish their actions are right now and that the future of the Open Web and HTML5 rides on them correcting this mistake.

  • Anonymous
    February 09, 2011
    @Leister - are you refering to Microsoft not implementing a Google owned proprietary video codec that is not in any way part of the HTML5 standard, or in any way or form even a standard? blogs.msdn.com/.../html5-and-web-video-questions-for-the-industry-from-the-community.aspx

  • Anonymous
    February 09, 2011
    @Jane: There is not one "Google owned proprietary video codec" in this world, not one that I know of. When you make a statement and still want to look good, please provide proof. No one likes someone who talks about things that are downright wrong. See en.wikipedia.org/.../WebM for a start. Also, be as irresponsible as you want when posting a comment, but for the sake of common sense, do not quote an MSDN article when arguing for Microsoft. It's the most unconvincing way of doing citations there is.

  • Anonymous
    February 09, 2011
    Your statement about the "When can I use" site using "Feature Existence" to test support is incorrect. All support is based on tests done by humans, though they are indeed not Standards Compliance tests. For example, based on automated tests contenteditable is supported on most mobile browsers, but for practical purposes it just doesn't work, and the "When can I use" site reflects this correctly. Thus it would be much more correct to list the site under "Compatibility Charts". Thank you.

  • Anonymous
    February 09, 2011
    @YANG: How can Google determine the spec and licensing of VP8 if they don't own it? The spec even point to Google source code as the defining standard if in conflict with published bitstream spec.(!) I know they have open sourced it and that it is free (given no patent backlash), that doesn't make it a standard, or out of Google control. It is not adopted or controlled by any standard organization. The link you obviously didn't bother to read wasn't to a MSDN article, but to a user comment.

  • Anonymous
    February 09, 2011
    The comment has been removed

  • Anonymous
    February 14, 2011
    http://www.partinchina.com http://www.hqew.net

  • Anonymous
    February 15, 2011
    The comment has been removed

  • Anonymous
    February 15, 2011
    The comment has been removed

  • Anonymous
    February 15, 2011
    What I want to know is ... how do you know what I may or may not want to use in a web application in the future? Web developers would love to use offline storage but don't because one of the major browsers doesn't support it. And it still won't so oh well, a great feature I would love to use but i can't. Microsoft isn't so much trying to support what developers WANT to use but forcing developers to use what Microsoft SUPPORTS. Big difference and one I don't like being shackled with.

  • Anonymous
    February 15, 2011
    people.mozilla.com/.../ie9 people.mozilla.com/.../ie9_vs_fx4.html Serious guys, why don't you just give up?

  • Anonymous
    February 16, 2011
    if ( IE ) return false;

  • Anonymous
    February 16, 2011
    Microsoft, you're thankfully slowly becoming more and more irrelevant. Why don't you, instead of keeping with your old tactics of attempting wherever possible to force people to things your way, instead embrace open standards? Web developers will thank you. Users will thank you, though they won't know they are - they'll like the new functionality though. And your own web-team will thank you too. I can't even imagine the scowls the Microsoft WebDev team give to the IE team as they walk past each other in the corridor. Also, even better - why not use Webkit in IE? Maybe us developers might laugh at you a little, but the general public wouldn't know the difference (besides speed and new functionality) and hey - we'd stop laughing after a bit :)

  • Anonymous
    February 16, 2011
    Microsoft, you're thankfully slowly becoming more and more irrelevant. Why don't you, instead of keeping with your old tactics of attempting wherever possible to force people to things your way, instead embrace open standards? Web developers will thank you. Users will thank you, though they won't know they are - they'll like the new functionality though. And your own web-team will thank you too. I can't even imagine the scowls the Microsoft WebDev team give to the IE team as they walk past each other in the corridor. Also, even better - why not use Webkit in IE? Maybe us developers might laugh at you a little, but the general public wouldn't know the difference (besides speed and new functionality) and hey - we'd stop laughing after a bit :)

  • Anonymous
    February 16, 2011
    The comment has been removed

  • Anonymous
    February 16, 2011
    people.mozilla.com/.../ie9

  • Anonymous
    February 16, 2011
    Yeah, Tim, it's because teh site is written in .NET, which aside from IE being the worst browser, .NET is the worst thing to ever happen to web development. Yes, Microsoft has done it twice, congrats!

  • Anonymous
    February 19, 2011
    The comment has been removed

  • Anonymous
    February 22, 2011
    what this guy said, word for word ------^^^^^^