Jaa

Using Hardware to Decode and Load JPG Images up to 45% faster in Internet Explorer 11

  • Artikkeli

Internet Explorer 11 and Windows Store Apps on Windows 8.1 offload parts of the image decoding pipeline to the graphics hardware, resulting in up to 45% faster image load, up to 40% lower memory consumption, and improved battery life. Images on average account for the most bytes downloaded on the Web today. To improve the performance of loading JPG images, IE11 has streamlined its JPG decoding by moving some steps of the decoding process directly to the GPU where it can be done significantly faster and in parallel.

JPG Image Format

Images account for 61% of bytes downloaded on the Web today, and 47% of image requests are for JPG images. By using hardware more efficiently to decode JPG images, IE11 now loads JPG images up to 45% faster and uses up to 40% less memory over previous IE versions.

To understand these performance improvements, let’s start by looking at how JPG images are typically encoded. The first step of JPG encoding is to convert the bitmap from the RGB color space to YCbCr color space.

Image of the JPG Encoding Process

RGB defines color in three color components: red, green, and blue. If we break down this image of the Grand Tetons in to its RGB color channels, you can see that there is a high degree of detail in each of the channels. The amount of memory required by IE for an image in the RGB color model can be calculated by multiplying image width x image height x 32 bits.

Image of the Grand Tetons with its Red, Blue, and Green components

Image of the Grand Tetons with its Red, Blue, and Green components

YCbCr (commonly referred to as YUV color space) defines a color space in terms of one luma (Y) and two chroma (CbCr) components. The luma component represents the brightness information of a color. The chroma components contain the color differences, with Cb containing the blue difference and Cr containing the red difference. The figure below shows the same image of the Grand Tetons broken down into its Y, Cb, and Cr channels.

Image of the Grand Tetons with its Y, Cb, Cr channels

Image of the Grand Tetons with its Y, Cb, Cr channels

The next step in the encoding process is to compress the image size using a lossy compression known as chroma subsampling. The chroma channels can be significantly compressed because the human eye is more sensitive to the brightness of an image and less sensitive to color or hue. For example, looking at the Cb and Cr channels, the last two on the right, you can see that there is very little information contained in these channels relative to the Y channel. The level of subsampling is typically expressed using a three part ratio, e.g., 4:2:0. The first part of the ratio represents a horizontal sampling reference, and the second two parts represent the number of chrominance samples in the first and second rows of that sampling reference. Chroma is typically subsampled at one of three levels:

  • 4:4:4, where the chroma data isn’t subsampled
  • 4:2:2, where the chroma data is downscaled horizontally by 2,
  • 4:2:0, where the chroma data is downscaled horizontally and vertically by 2.

A 4:2:0 subsampled YCbCr JPG image can use up to 62.5% less memory than the original RGB bitmap! Most JPG images used on the Web today are already in 4:2:2 or 4:2:0 chroma subsampling modes. Most image processing tools will automatically subsample to 4:2:2 or 4:2:0 when using the ‘Save for Web’ option.

Once the image has been subsampled, the discrete cosine transformation, quantization, and Huffman encoding processes are also applied to get the final encoded JPG image.

More Efficiently Using Hardware in IE11

To decode a JPG image, you run the same encoding steps in reverse. Traditionally, IE had always decoded JPG images into RGB bitmaps by running all of the below decoding steps directly on the CPU. At render time, IE would copy the RGB bitmap to the GPU for rendering.

Image of the JPG Decoding Process

To more efficiently use the hardware, IE11 now splits the JPG decoding work between the CPU and GPU. IE11 decodes the JPG image into the chroma subsampled YCbCr color space on the CPU, but then does the chroma upsampling and YCbCr to RGB color conversion steps on the GPU at draw time, where it can happen much faster and in parallel. This process frees CPU time to perform other operations, as the CPU is a common bottleneck in modern sites and apps. In addition to the decode time improvements, copying the much smaller YCbCr image to the GPU reduces the amount of memory that is copied and stored on the GPU (a limited resource). Using less CPU and memory also reduces power consumption and increases data locality.

The below CPU chart shows the amount of time it takes to decode and draw a 4:2:0 subsampled YCbCr JPG version of the Grand Tetons image on IE10 on Windows 8. Image decoding took 31.9ms, and total time to draw the image was 81.5ms.

The below CPU chart shows the amount of time it takes to decode and draw a 4:2:0 subsampled YCbCr JPG version of the Grand Tetons image on IE10 on Windows 8. Image decoding took 31.9ms, and total time to draw the image was 81.5ms.

If we load the same JPG image on IE11 on Windows 8.1, you can see that the decode time is now only 17.9ms, which is an improvement of 44%! The total time to draw the image is now 57.5ms, which is 30% faster.

If we load the same JPG image on IE11 on Windows 8.1, you can see that the decode time is now only 17.9ms, which is an improvement of 44%! The total time to draw the image is now 57.5ms, which is 30% faster.

Because most JPG images on the Web today are in YCbCr format, IE11 on Windows 8.1 users will experience these improvements automatically. We recommend that developers ensure their JPG images are compressed in 4:2:2 or 4:2:0 chroma subsampling modes to maximize the performance benefits of hardware accelerating the JPG decoding pipeline.

Summary

By using the hardware more efficiently to decode JPG images, IE11 improves the performance of nearly every page you browse, while also improving the power consumption and battery life of your device. Please install the Windows 8.1 Preview from the Windows Store and try IE11. As always, we welcome your feedback, either through the IE11 Send Feedback tool or on Connect.

Jatinder Mann, Internet Explorer Program Manager

Comments

  • Anonymous
    September 12, 2013
    It's nice to see GPU acceleration so prevalent; but are there any chances that you can improve picture resizing as well? It was a long standing problem on many browsers, but fixed somewhat on them. Basically, pictures are very aliased when shrinked down. Testcase: bug486918.bugzilla.mozilla.org/attachment.cgi

  • Anonymous
    September 12, 2013
    The comment has been removed

  • Anonymous
    September 12, 2013
    @Fleet Command: Yeah, I can't wait to use all those new cross-platform browser features, like -webkit-mask-image. I also can't wait for websites to stop using -webkit-gradient and -webkit-transform and -webkit-transition and -webkit-any-other-css3, so they can be fully functional on my css3-compliant browsers.

  • Anonymous
    September 12, 2013
    How about you just finish the WebGL 1.0 implementation, so we can view Nokia Here maps! connect.microsoft.com/.../nokia-here-maps-reporting-webgl-issues And please support APNG on IE connect.microsoft.com/.../apng-support Thank you.

  • Anonymous
    September 12, 2013
    @Fleet Command: yeah, like all other browsers are not fighting to shave extra milliseconds here and there. At this point, I think trying to optimize image rendering performances will bring more benefits to users than optimizing JavaScript for certain benchmarks. Not that trying to optimize JS is bad, but there are probably more low-hanging fruits when it comes to the GPU.

  • Anonymous
    September 12, 2013
    IE11 is going to be on Windows 7 too, although it might not include this change.

  • Anonymous
    September 12, 2013
    This change is actually nice. That's what graphics cards are for: to deal with graphics. Still, I wouldn't mind if the version for Win 7 had aero style restored. This grey scroll bar is hideous and really hard to see on most websites. The person who came up with it should be tarred 'n' feathered.

  • Anonymous
    September 12, 2013
    fufj

  • Anonymous
    September 12, 2013
    > "We recommend that developers ensure their JPG images are compressed in 4:2:2 or 4:2:0 chroma subsampling modes" How can I check this in Photoshop (or other applications) ?

  • Anonymous
    September 12, 2013
    I'm actually impressed by this post.  This is the first post I've read on the IE blog where there is a real gain for the end user (and as such the developers). No proprietary IE-only code, no non-standards garbage... just pure Engineering Might put to work! - Well Done! PS Mozilla, Google, Opera, Apple... cough, cough... get on this!

  • Anonymous
    September 12, 2013
    @ZippyV: Your answer is in the text: "Most image processing tools will automatically subsample to 4:2:2 or 4:2:0 when using the ‘Save for Web’ option." "Save for Web" is a Photoshop command by the way. I don't remember any other tool with this command. @Stifu: Maybe they do. I don't blame them, since they also work on major changes I mentioned earlier. But IE team works exclusively on minor changes. And anyway, those who get these features are very few. Compare Windows 8.1 users (IE11) vs. users of Windows, Mac OS X, Linux and Android. (Firefox, Chrome and Maxthon)

  • Anonymous
    September 12, 2013
    @Fleet Command: alright, I'll go ahead and feed the troll. It seems hungry, and I can't let it starve like that. "But IE team works exclusively on minor changes" So, adding WebGL support is a minor change, for example? The same goes for HTML5, CSS3, SVG, GPU acceleration, JITed JS, and all that, yeah? Your only argument against IE seems to be that it's not cross platform. Yeah, indeed, you got that right. But tell me more about the "major changes" other browser makers are working on, which puts them in a different league. "And anyway, those who get these features are very few. Compare Windows 8.1 users (IE11) vs. users of Windows, Mac OS X, Linux and Android. (Firefox, Chrome and Maxthon)" Even if IE11 was for Windows 8.1 only (spoiler: it's not), the target OS is completely irrelevent with the discussion here. Sometimes there are things to *** about, but not here nor now. The IE team is working on making their browser better, like other browser makers. The change detailed in this article is nice, and other browser makers should take note of it. End of story.

  • Anonymous
    September 13, 2013
    So no such acceleration for 4:4:4 images or will there simply be no difference to IE10?

  • Anonymous
    September 13, 2013
    @Blurga: The post says that YCbCr to RGB conversion is offloaded to the GPU, not just upsampling, so it should still work faster. Also with 4:4:4 images you don't need to upsample anything to begin with, anyway... The recommendation to use 4:2:2 is probably to get smaller jpg image sizes.

  • Anonymous
    September 13, 2013
    @Fleet Command: All progres is welcome, at least the IE team works on making IE faster instead of adding features that shouldn't be in a browser (by default). Beside, since when is full GPU acceleration, WebGL, HTML5, CSS3, etc. a minor feature. Another big change in IE is the new interface in Internet Explorer9, the Internet Explorer 10 app and the redesigned and improved Internet Explorer 11 app.

  • Anonymous
    September 13, 2013
    Nice to see. What you really should do is:

  1. Chrome, WebKit and Firefox are Open Source projects, go there and support for JPEG-XR. It's time to say goodbye to JPEG. We really need 10Bit support and better compression algorithm.
  2. Add APNG support. The GIF is too limiting. Thanks.

  • Anonymous
    September 13, 2013
    So, is the image decoded on the GPU and then pulled back as RGB to be repushed as RGB over and over, saving a grand total of what, 500ms off a page load?  or does the JPEG have to be repushed to the GPU with ever render meaning it has to be continually redecoded over, and over and over and over meaning time usage and more battery usage over time? Frankly, this seems like a seriously stupid investment of effort no matter how you look at it.

  • Anonymous
    September 13, 2013
    libjpeg-turbo.virtualgl.org sourceforge.net/.../libjpeg-turbo Firefox uses libjpeg-turbo 1.3.0 and is 2x-4x(200%-400%) faster in JPEG decoding. Don't be fooled by MS, they're outgunned and outperformed by superior open source software and code.

  • Anonymous
    September 14, 2013
    @ Open Source Libjpeg-turbo and what IE11 is doing are two different things. Libjpeg-turbo is a codec while what IE11 is doing is using the GPU to decode JPEGs.

  • Anonymous
    September 14, 2013
    @Royi A: According to caniuse.com, IE9/10/11 is, in fact, the ONLY browser to support JPEG XR.

  • Anonymous
    September 14, 2013
    @Arieta @Royi : Because Microsoft invented it.

  • Anonymous
    September 14, 2013
    @Royi A @Arieta @Heath Firefox developers looked into adding JPEG-XR and were pretty sold on using it. But Microsoft has not released any Open Source JPEGXR codecs that they could use. No one else in the OSS community has bothered to do so either. WebP gets all the attention these days. If Microsoft were ever committed to the stuff they release into the wild, they would make a decoder that OSS projects can use cross platform.

  • Anonymous
    September 14, 2013
    After reading these "Not enough!" comments we can conclude that .. Instead of only optimizing performance of something which people didn't asked for, if you guys added support for APNG and related formats, the reaction would have been different from. For example, its being a while there is no blog from SVG team. SVG has some unresolved issues since IE 9. Compare this svg tavmjong.free.fr/.../svg_test.svg  in FF, IE and Chrome. There is one significant issue with incorrectly inheritance of fill opacity filter (connect.microsoft.com/.../wrong-inheritance-with-svgs-fill-opacity). As IE team has closed this issue as "Won't Fix" and we can't ask for ETA.. I see no value in submitting bugs on connect.

  • Anonymous
    September 15, 2013
    The comment has been removed

  • Anonymous
    September 15, 2013
    @Royi A: you seem to be stuck in 2008 or something. Read up on WebP.

  • Anonymous
    September 15, 2013
    @Reality: Microsoft released JPEG XR under a BSD License earlier this year, and an open source library has also been released on CodePlex in April; see http://jxrlib.codeplex.com

  • Anonymous
    September 16, 2013
    @BlueInk - Great. JPEG XR support was added to Internet Explorer 9. So, around three years have passed since the first developer preview of Internet Explorer 9, until they released that open source library in April (according to you, I have not checked). I guess they did not care about the format enough to convince developers to use it (I never heard or saw anything like 'move to JPEG XR' or any other promotion of the format), or other browser vendors to implement it.

  • Anonymous
    September 16, 2013
    I love all of the hardware accelerated things that IE has been doing in the last few releases, I really do. It does create a problem though, when an average user goes to a website in IE and the website is "broken". Then they go to the website in "Google" or "FoxFire" (or however they mis-remember their other browser names) and the website works. So of course the problem is that IE is broken. But when a technical user digs into the issue and finds out that the problem is their outdated graphics card driver, and updating it fixes IE, the user would still blame IE. IE needs some sort of visual detection and hinting that graphics card may be the source of problems and hopefully shim out broken combinations. There should be more granular options for software rendering instead of hardware accelerated rendering to test why only certain websites are broken in IE, on specific computers.

  • Anonymous
    September 16, 2013
    Apologies for my last comment. After seeing this link: ie.microsoft.com/.../ImageSupport I realized that I am a sick.. seriously. I don't know what's my problem.. I always find myself pulling some sort of rant.. if they don't support some standard, I will b-itch, if they do i'll b-itch about something else.. All I want is to see IE fail miserably but the recent developments and improvements are against my will. Damm.. DIE IE DIEEE!!!! can't help myself! it has been always like this, it will always be the same.

  • Anonymous
    September 16, 2013
    @Xero @To Recapitulate   Thanks for supporting us and sending us feedback. We've reactivated the feedback for consideration in future releases. Thanks again, Jay Hong [MSFT]

  • Anonymous
    September 16, 2013
    @Jay Hong [MSFT]: it's nice to see you care about user feedback, but seriously, APNG is dead in the water. It has been refused for standardization by the PNG group, and is only pushed by Mozilla (they implemented it 5 years ago, and no one followed). APNG couldn't survive that stupid war with MNG. Anyway, WebP does more things and has more backing.

  • Anonymous
    September 16, 2013
    @Phistuck - Cute.

  • Anonymous
    September 16, 2013
    @Jay Hong [MSFT], Thanks for WebGL. Please consider WebRTC request. connect.microsoft.com/.../ie11-feature-request-support-for-webrtc The world is watching and developers are waiting for Microsoft to make a move: stackoverflow.com/.../1712065

  • Anonymous
    September 17, 2013
    @Jay Hong [MSFT], thanks for the consideration. It feels so amazing when we get any kind of response from IE team! :-) By any chance, if you know anyone in IE's printing team, can you please escalate these two bugs with them:

  • Anonymous
    September 17, 2013
    The comment has been removed

  • Anonymous
    September 17, 2013
    The comment has been removed

  • Anonymous
    September 17, 2013
    Sorry for the off-topic question but isn't the "window.external.msReportSafeUrl" insecure? Imagine visiting a malicious site and reporting itself that it is safe to Microsoft and then Smartscreen Filter trusting it. Shouldn't the user only be able to use this feature? msdn.microsoft.com/.../hh848317(v=vs.85).aspx Also why is this API returning a HRESULT? What is a HRESULT anyway? In JavaScript there is no such type. Isn't it?

  • Anonymous
    September 17, 2013
    Thanks for the positive feedback on Connect. The team's been working hard to make it a more effective feedback mechanism, and we know that there's still more for us to do. As I've mentioned in some previous posts we're investigating what it will take to fix the blog software. We know that the "lost comment" bug goes away if we disable anonymous posts, but we'd like to avoid that if we can. In the interim, if you sign in you shouldn't hit the bug.

  • Anonymous
    September 17, 2013
    The comment has been removed

  • Anonymous
    September 17, 2013
    The comment has been removed

  • Anonymous
    September 17, 2013
    Test comment

  • Anonymous
    September 17, 2013
    @Real McCoy So HRESULT is cast into which JavaScript object exactly? String, number, what? And if it is cast to some JavaScript object, then thy doesn't MSDN say that this particular type is the return type instead of talking about HRESULT? Secondly, so you are saying that if this API is called by JavaScript code, then the user will be prompted? Are you sure about that? I thought that the API silently reports the current page URL as safe. The MSDN topic didn't say anything about user consent.

  • Anonymous
    September 17, 2013
    The comment has been removed

  • Anonymous
    October 03, 2013
    The comment has been removed