DWM's use of DirectX, GPUs, and hardware acceleration

For the last few years, both desktop and laptop PCs have been outfitted with increasingly powerful graphics chipsets, including blazingly fast geometry and pixel processing, higher fill rates, and faster and faster bandwidth between system memory and video memory.  Further, GPU computational increases has been exceeding Moore's Law for some time, outpacing CPU growth.  The beneficiaries of this growth has almost entirely been PC games and the gaming community.  However, this processing power has largely sat dormant during typical use of the Windows desktop and typical productity/business/everday applications.

One of the key goals of the Desktop Window Manager in Windows Vista is to harness all this spare graphics compute power, and bring it to bear on making the computing experience better for the majority of users who are not gamers, or at least not always gamers.

In order to do this, the DWM is built upon Microsoft DirectX, specifically Direct3D, which is the uniform way that software in the Windows environment talks to graphics hardware.  More precisely, the DWM is actually built directly atop a layer we internally call "milcore", and milcore is built directly atop DirectX.  (More on milcore in a future post.)  But the end result is that Direct3D textures are created that represent the window content and the window frame, and DWM/milcore is responsible for issuing the proper Direct3D calls that result in composing all of the Direct3D textures to form the desktop.

One specific takeaway from this: the Windows Vista desktop (through the DWM) is a fullscreen Direct3D application.

Fundamentally 3D

You may know, though, that in typical usage of Direct3D, one doesn't directly bit-blit textures to a primary surface.  This is also true in the DWM.  The window contents aren't being asked to paint directly to the desktop surface.  Instead, they're being treated as texture maps on a 3D mesh that is being rendered by Direct3D and the graphics hardware.  For example, in the case of the standard desktop view, it so happens that the 3D mesh that the textures are being mapped to is a very simple rectangle (actually, two adjacent triangles), representing the client area of the window. 

This is important, because once that's seen as being the case, it's not a far leap at all to understand how the DWM does it's 3D window transitions, or the 3D "Flip3D" view as shown in this initial DWM post.  Our window transitions (open, close, minimize, restore), are simply 3D transformations of that rectangle that already exists in 3-space.  It would not be much harder to modify any of these transitions to be more complex - for instance to map the window content onto a spinning cylinder for minimize.  Thankfully we have a crack team of graphic and UI designers that prevent us from making such UI faux pas. 

Pixel Shaders for Blur

One of the signature looks for Windows Vista is the window border's "glass" look.  This look allows one to see through the border to what's behind, but also adds some amount of translucency and also blur to what's behind.  From a design perspective, this makes window borders feel less heavy and in-your-face, and let's them more readily do their job without calling attention to themselves. 

Translucency is readily achievable in most graphics systems (via the alpha channel) and really isn't such a big deal.  However, even though it's not completely transparent, windows borders that only have translucency do not provide a good UI experience.  That's because the content behind the borders is too clear and discernible, and distracts the user from the window they're manipulating.

Therefore, a key element in the Aero design is the blurring of content behind the "glass" borders, to avoid that content being too clear and discernible.  This blur is surprisingly difficult to achieve, and the DWM team has spent a lot of brainpower and effort getting it to where it is:

• First, there is the actual blurring of the underlying content.  This is done by a custom PS 2.0 pixel shader.  Pixel shaders are small programs that run directly on the GPU, which acts essentially as a SIMD machine able to process many pixels in parallel.
• The source of the blurring must be a partially composited desktop - everything behind the window border that's being rendered, but not the window border or what's in front of it.  And that needs to be pulled out into a different buffer.
• Next, the way blur works is by taking neighboring pixels and averaging them together to create a resultant pixel.  This means that when a window with "glass" moves, it's not sufficient to just re-render what's behind it.  The area re-rendered needs to be extended out a little bit (by the radius of the blur).  Without care, this extension can accumulate across multiple windows and lead to invalidating larger and larger portions of the screen.

All of these combine to create the smooth, seamless blur that Aero glass exhibits; and does so smoothly and quickly so the user doesn't perceive any hiccups on window motion or when "glass" is in front of updating scenes, like video playing.

Flipping and Tearing

Modern DirectX applications, on modern DirectX 9 class graphics chipsets (which are a requirement for running the Windows Vista DWM), often use "flipping" in order to present their content.  Flipping is when the video card is able to present video memory surface A to the monitor while the application writes to video memory surface B.  When the application is done with its update, it schedules a "flip" through DirectX, which is executed at the next monitor vertical blank.  When executed, the video card now starts reading out from surface B, and the application is freed to write into surface A.  On the next flip, they swap back. 

The result of this is that the user perceives exactly what the application wants it to without any of the "tearing" that frequently accompanies applications writing directly to the buffer that's being displayed when a screen refresh happens in the middle of the application's update process.

The DWM uses exactly this technique to avoid window tearing during refresh, and this results in a much smoother desktop.

But Isn't There a Limit to Video Memory?

There are limits to how much video memory an application can use, and how much GPU time is available.  So how can a general window system operate within these limits when there are no limits on, say, the number of windows that can be opened?  Answer: the Windows Display Driver Model (WDDM, aka LDDM) makes all of this viable.  How's that?  That'll be the topic for my next blog entry on the DWM.

Comments

• Anonymous
  March 19, 2006
  Of course, some of our customers have certain cylindrical-wrapping needs . . . but I guess that's our job, not yours. :-)

• Anonymous
  March 19, 2006
  The comment has been removed

• Anonymous
  March 19, 2006
  greg, as i know at this point a directx8 card is able to render glass ( in current CTPs). does that mean a different kind of glass rendering(the PS 2.0 version) is coming?
  
  i have a second question: why isnt it possible for the DWM to work with software rendering? i'm curious just because OS X has similar functionality since 2001, and back then there weren't any directx 9 cards out there.
  i think DWM is one of the best things in Vista (THE thing for me, i'm going to upgrade just for this feature alone, i'm sick of the window drawing artifacts XP has), and a lot of people on laptops for example, without an upgrade path, will be locked out of this world of compositing. they could probably live without glass, but compositing should work with lower hardware requirements as well.

• Anonymous
  March 19, 2006
  Thanks for doing this blog. :) Its of great interest for me as I love the new DWM interface and hope to employ glass in some of my future programming.
  
  As a suggestion for some additional topics to cover for this blog post series, could I ask if you could maybe go in to a little more detail on some of the performance issues that came up during development and how they were solved to keep Windows running smooth?
  Also, while I only know a little about it myself, since DWM appeared, Linux programmers have been busy and whipping up their own 3D thing called XGL where their multiple desktops are rendered on a cube and windows get fancy effects too. While I'm not asking you to put it down, I'd be curious to know how DWM and XGL differ (other than one is DirectX based and one is OpenGL based) and what the key benefits of DWM are beyond simple appearance. (As I get moaned at all the time by Linux friends who say its nothing but sparkle!)

• Anonymous
  March 20, 2006
  i would also like an option to create custom transition effects or more fancier transition effects. you know at which competitor to look ;)
  
  another question: will there be more animations and transitions inside the vista gui, not only in the desktop manager? or are those windows like explorer in vista again limited to the slow GDI+ that doesn't allow sleek transitions [like transitions when switching between folders or folder views in explorer or panels animating in and out]?

• Anonymous
  March 20, 2006
  If an app is using the hardware for rendering, how to you ensure the DWM doesn't conflict with the app for use of the hardware?
  - From a scheduling point of view, couldn't a bad app hog the GPU, preventing the DWM from running?
  - From a memory point of view, what happens if an app has locked its target surface and the DWM wants the contents of it?

• Anonymous
  March 25, 2006
  My earlier post on "DWM's use of DirectX, GPUs, and hardware acceleration" generated some good comments...

• Anonymous
  March 30, 2006
  The comment has been removed

• Anonymous
  March 30, 2006
  I would like to know if we can drag and drop the tasks running in the taskbar around to put certain tasks in certain order all the time. This is something that I have wanted for some time, it is a personalation feature that would allow me to more consistantly operate the computer.

• Anonymous
  April 07, 2006
  Being a Gentoo Linux user myself, I'm surprised how many times I've seen the word Linux in this, blog.  Not bad considering their advertising budget.  But I know "my win friends", here don't get it.  "It" being the thousands of hours
  being put into linux development everyday.  
  
  Besides the "sparkle" and I know the average home
  computer user, uses their computer for strictly entertainment purposes.  The more bling the better.  That is exactly why, you heard it here first, "Xgl and linux will be a historical watershed for linux." Not that this technology is much differnt than described here.  The "Linux noobs" appearing in Gentoo is amazing, for alpha software?  Already "Plugins" are being developed faster than I can keep up.  1000's of web pages can be found since the release of these technologies by Suse in Jan.
  By the time this technology comes to stable, "go figure" probably about the same time as Vista, there will probably be 100+ plug-ins for
  "sparkle".  
  
  But don't worry there won't too much lose to win users.  10-100 million in advertising you probably wont lose more than 1-2% of win users.
  
  Thats a watershed event to me.
  
  Ready, get set, ..

• Anonymous
  April 17, 2006
  The comment has been removed

• Anonymous
  April 21, 2006
  Here's a list of topics that I have posted on (with active links) or expect to post on (without links)...

• Anonymous
  June 07, 2006
  When talking about WPF during the Windows Vista ISV Touchdown training a lot of people were interested...

• Anonymous
  May 31, 2007
  When talking about WPF during the Windows Vista ISV Touchdown training a lot of people were interested

• Anonymous
  May 26, 2009
  PingBack from http://backyardshed.info/story.php?title=greg-schechter-s-blog-dwm-s-use-of-directx-gpus-and-hardware

• Anonymous
  May 30, 2009
  PingBack from http://outdoorceilingfansite.info/story.php?id=2173

• Anonymous
  May 31, 2009
  PingBack from http://woodtvstand.info/story.php?id=581

• Anonymous
  June 01, 2009
  PingBack from http://uniformstores.info/story.php?id=15760

• Anonymous
  June 17, 2009
  PingBack from http://patiosetsite.info/story.php?id=657