Windows Vista Sound causes Network Throughput slowdowns.
AKA: How I spent last week :).
On Tuesday Morning last week, I got an email from "reader@slashdot.org":
You've probably already seen this article, but just in case I'd love to hear your response.
https://it.slashdot.org/article.pl?sid=07/08/21/1441240
Playing Music Slows Vista Network Performance?
In fact, I'd not seen this until it was pointed out to me. It seemed surprising, so I went to talk to our perf people, and I ran some experiments on my own.
They didn't know what was up, and I was unable to reproduce the failure on any of my systems, so I figured it was a false alarm (we get them regularly). It turns out that at the same time, the networking team had heard about the same problem and they WERE able to reproduce the problem. I also kept on digging and by lunchtime, I'd also generated a clean reproduction of the problem in my office.
At the same time, Adrian Kingsley-Hughes over at ZDNet Blogs picked up the issue and started writing about the issue.
By Friday, we'd pretty much figured out what was going on and why different groups were seeing different results - it turns out that the issue was highly dependent on your network topology and the amount of data you were pumping through your network adapter - the reason I hadn't been able to reproduce it is that I only have a 100mbit Ethernet adapter in my office - you can get the problem to reproduce on 100mbit networks, but you've really got to work at it to make it visible. Some of the people working on the problem sent a private email to Adrian Kingsley-Hughes on Friday evening reporting the results of our investigation, and Mark Russinovich (a Technical Fellow, and all around insanely smart guy) wrote up a detailed post explaining what's going on in insane detail which he posted this morning.
Essentially, the root of the problem is that for Vista, when you're playing multimedia content, the system throttles incoming network packets to prevent them from overwhelming the multimedia rendering path - the system will only process 10,000 network frames per second (this is a hideously simplistic explanation, see Mark's post for the details)
For 100mbit networks, this isn't a problem - it's pretty hard to get a 100mbit network to generate 10,000 frames in a second (you need to have a hefty CPU and send LOTS of tiny packets), but on a gigabit network, it's really easy to hit the limit.
One of the comments that came up on Adrian's blog was a comment from George Ou (another zdnet blogger):
""The connection between media playback and networking is not immediately obvious. But as you know, the drivers involved in both activities run at extremely high priority. As a result, the network driver can cause media playback to degrade."
I can't believe we have to put up with this in the era of dual core and quad core computers. Slap the network driver on one CPU core and put the audio playback on another core and problem solved. But even single core CPUs are so fast that this shouldn't ever be a problem even if audio playback gets priority over network-related CPU usage. It's not like network-related CPU consumption uses more than 50% CPU on a modern dual-core processor even when throughput hits 500 mbps. There’s just no excuse for this."
At some level, George is right - machines these days are really fast and they can do a lot. But George is missing one of the critical differences between multimedia processing and other processing.
Multimedia playback is fundamentally different from most of the day-to-day operations that occur on your computer. The core of the problem is that multimedia playback is inherently isochronous. For instance, in Vista, the audio engine runs with a periodicity of 10 milliseconds. That means that every 10 milliseconds, it MUST wake up and process the next set of audio samples, or the user will hear a "pop" or “stutter” in their audio playback. It doesn’t matter how fast your processor is, or how many CPU cores it has, the engine MUST wake up every 10 milliseconds, or you get a “glitch”.
For almost everything else in the system, if the system locked up for even as long as 50 milliseconds, you’d never notice it. But for multimedia content (especially for audio content), you absolutely will notice the problem. The core reason behind it has to do with the physics of sound, but whenever there’s a discontinuity in the audio stream, a high frequency transient is generated. The human ear is quite sensitive to these high frequency transients (they sound like "clicks" or "pops").
Anything that stops the audio engine from getting to run every 10 milliseconds (like a flurry of high priority network interrupts) will be clearly perceptible. So it doesn’t matter how much horsepower your machine has, it’s about how many interrupts have to be processed.
We had a meeting the other day with the networking people where we demonstrated the magnitude of the problem - it was pretty dramatic, even on the top-of-the-line laptop. On a lower-end machine it's even more dramatic. On some machines, heavy networking can turn video rendering to a slideshow.
Any car buffs will immediately want to shoot me for this analogy, because I’m sure it’s highly inaccurate (I am NOT a car person), but I think it works: You could almost think of this as an engine with a slip in the timing belt – you’re fine when you’re running the engine at low revs, because the slip doesn’t affect things enough to notice. But when you run the engine at high RPM, the slip becomes catastrophic – the engine requires that the timing be totally accurate, but because it isn’t, valves don’t open when they have to and the engine melts down.
Anyway, that's a long winded discussion. The good news is that the right people are actively engaged on working to ensure that a fix is made available for the problem.
Comments
Anonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
Thanks for posting this information. I'm sure a tested fix will take some time but it's good to just have an explanation so that people can't spread ridiculous, baseless FUD about how the problem is caused by DRM and so on. (I'm no fan of DRM but I dislike the wrong thing being blamed for a problem.)Anonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
PingBack from http://msdnrss.thecoderblogs.com/2007/08/28/windows-vista-sound-causes-network-throughput-slowdowns/Anonymous
August 28, 2007
Nik: It shows up while copying files around, and on a gig network it matters. Michael: Because that's not the way the hardware works. On multicore machines, the hardware interrupts are only serviced on processor 0. Even on true MP machines, many of them only service interrupts on processor 0. There's nothing an operating system can do to fix it. However there ARE some things that you can do to mitigate the issue, and the right people are working on creating the fix.Anonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
@ Nik: Although one my think this to be true, in the broadcasting industry 'Audio servers' keep an online cache of around 100GB of 24 bit WAV audio that normally resides in a tape or MAID Archive. I won't do the maths but a typical broadast audio server has 16 outputs, each under seperate control for a different TV channel. Voice overs, Audio descrption services, mutiple languages for the same program, etc,etc. Getting 16 channel of all this content in 24bit WAV files from an archive is pretty intensive, and although we still use WinServer2003, this particular issue could have big consequences for us if we moved to Vista, or even WinServer 2008! Anyway, why doesn't this hard coded value ramp up with CPU speed? Could this value be an output of the windows Experience index? SteveAnonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
"And anybody playing multimedia will not need massive networking..." I have to wonder if that is true. What if one has a Vista MCE computer with HDTV tuners, able to watch and record HDTV, and then what if there are other computers on a gigabit network simultaneously watching (aka streaming) recorded HDTV off that same Vista computer. Seems there would be a very common need to support both a heavy network requirement and multimedia playback.Anonymous
August 28, 2007
WhatAboutHD: By our measurements, you can run at least 2 HD 1080P video streams over the network without encountering this issue. This really is limited to file copies or other hideously network intensive operations.Anonymous
August 28, 2007
Larry: I got the impression from Russinovich's blog post that the problem tends to be much worse in a computer with multiple network adapters (ie 10.000 packets/sec becomes 6.000 packets/sec with three adapters). I may be an extreme example but I've got 7 adapters on my machine. 2 VPN, a 1394, a LAN, a WLAN and two VMWare. Does this affect the speeds or does the adapters need to be in-use? I guess that 2 (or three if "1394" counts) is standard nowadays? I'm running Windows XP so I can unfortunately not try it out.Anonymous
August 28, 2007
August, it is. The VPN and 1394 don't count, but the vmware ones do. That's a part of the things we need to fix.Anonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
Thanks for the response. I guess I misunderstood. I thought Mark's post was saying MMCSS threads may use up to 80%, leaving 20% for everyone else. I still hope the solution allows the user to control the tradeoff.Anonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
Here's a better explanation and how you can work around the issue with jumbo frames. http://blogs.zdnet.com/Ou/?p=711 The point is that the 10K packets per second rate limit is hard-coded for the worst case scenario. It does not account for faster multi-core CPUs. My Core 2 Duo E6400 could have easily been set to 50K packets per second and the same assurances to multimedia would have been guaranteed. The other problem is that MMCSS doesn’t distinguish between idle, simple music playback, DVD, or HD Video playback since those clearly have different processor requirements. This is a poor design and Microsoft will need to fix this and the simplest and most reasonable way to address this is with a content and CPU aware dynamic rate limit for networking performance. Making the rate limit per LAN interface and not amongst all the interfaces is probably a good idea too and that’s an obvious bug that needs to be fixed.Anonymous
August 28, 2007
OSGuy: I don't know how Linux handles it, I just know that the guys who know this kind of stuff over here tell me that multicore machines handle interrupts on CPU0. Some MP designs handle interrupts on separate processors, but the majority of the inexpensive (ie non server ones) just interrupt CPU0 - it's cheaper :). I can't speak to how Linux handles this, I'm not a kernel guy. George: I don't think that anyone is defending the decision to go with 10K packets/second. Certainly during the internal discussion of the problem (and I was on all of the emails) keeping the limit was never one of the suggestions tendered. Btw, I do like your jumbo frames idea, it's a good one. As Mark (and I) have said, we're going to address this issue.Anonymous
August 28, 2007
Larry - CPU0 is the default for handling interrupts, that's how the table is programmed initially - most modern OSes dynamically reprogram the APIC to distribute the interrupts if CPU0 is overburdened with handling interrupts and other CPUs are relatively idle. Anyway I assumed you were interested in knowing this, so apologies if that wasn't the case.Anonymous
August 28, 2007
George: Audio uses the "Audio" category (or the "Pro Audio" category, video playback uses the "Playback" category. There are many possible solutions to the problem, some of them have been mentioned on this thread, some of them aren't. The relevent teams have commited to providing a solution to the problem.Anonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
Larry, it's pretty sad that an OS 6 years in development, Microsoft's crown jewels is bested by XP. Really really sad and pathetic, in fact.Anonymous
August 28, 2007
It seems to me that the main culprit here is that the multimedia stack in Windows Vista is always geared towards low latency, whereas most of the time this isn't needed. For regular multimedia playback, I would actually prefer that the audio stack not try to maintain a 10ms mixing interval, because that burns more CPU time in context switches and disturbs video display timing (which can be critical in windowed display). For regular non-interactive playback, all you need is matched latencies between the audio and video streams. You can mix every 100ms and still have perfectly synced, glitch-free audio. A 10ms mixing rate also seems like a bad idea for regular desktop usage on laptops, where you want to lower tick rates so the CPU can sleep a bit. Any thought to making this adaptive or tunable in a future version?Anonymous
August 28, 2007
The comment has been removedAnonymous
August 28, 2007
10000 packets really isn't all that much if your packets are just 64 bytes. That's just 5 mbps. Well below some broadband speeds.Anonymous
August 29, 2007
George: This isn't a CPU utilization problem. If the multimedia processes were allowed to run at all, they'd be able to do their work. But in this scenario, without the throttling, under certain network workloads, the multimedia processes don't get scheduled for tens and hundreds of milliseconds. Which causes massive glitching on even non HD content. The types of playback don't actually matter.Anonymous
August 29, 2007
The comment has been removedAnonymous
August 29, 2007
A book on PBX systems (phone systems) once explained it like this. When dealing with data traffic, you need 100% accuracy but can tolerate some latency. When dealing with voice (i.e. sound), you can tolerate little or no latency but can degrade quality (more lossy compression or lower sampling bit sizes). The two are diametrically opposed. The Vista issue, is almost the exact same scenario playing out on a computer. (I just noticed Larry even mentioned voice communications)Anonymous
August 29, 2007
There is a workaround for this problem already. My previous roommate and friend found out you can work around it by removing a false service dependency: http://digg.com/microsoft/How_To_Fix_the_Vista_Network_Speed_Issue_While_Playing_SoundAnonymous
August 29, 2007
The obvious solution would be to give the ability to the running app to specify if it needs low latency or glitch-free audio.Anonymous
August 29, 2007
The comment has been removedAnonymous
August 29, 2007
The comment has been removedAnonymous
August 29, 2007
Still, I have zero problems playing audio or video while pushing or pulling 60 MB/sec or 480 mbps. It's obvious the throttling simply needs to be dynamic and take in to account how fast the user's CPU is.Anonymous
August 29, 2007
Doesn't this issue call for hardware audio/video implementation with local buffer/memory, with better DMA and driver models instead of last years trend to rely on host-based audio/video processing?Anonymous
August 29, 2007
The comment has been removedAnonymous
August 29, 2007
Sebastien: Hardware acceleration of audio wouldn't help this situation at all.Anonymous
August 29, 2007
I tested this in our office: gigabit connection between client and server with two gigabit switches in between. Downloading a large file from the server without WMP playing: about 40% usage of the gigabit NIC. Downloading a large file from the server with WMP playing: about 12% usage of the gigabit NIC. Obviously it's quite annoying when you have to move large files around often. However, this rarely happens while WMP is playing (it's an office, remember ;-) ).Anonymous
August 29, 2007
No it isn't obvious and I haven't seen the "slide show" effect even when I'm pulling in 400 mbps of data even when I'm playing DVDs. I don't even see a glitch when I'm playing back 10 videos at the same time. I don't doubt your Sr. Engineers, but maybe we're not communicating clearly here. I just don't see the problems on my hardware that you're describing.Anonymous
August 29, 2007
Last week there was a small storm on the internet when it was discovered that playing music on WindowsAnonymous
August 29, 2007
The comment has been removedAnonymous
August 29, 2007
The comment has been removedAnonymous
August 29, 2007
There is one thing I find funny about this whole "We've already got a fix." How many people internal to MS have been using Vista? How many people beta tested it? How many people have been using it since release? Now it is reported that there is a network slowdown. How many people does this issue really affect? But someone posts a "fix" and 20-30 people say it worked for them but a few said it made the issue worse. But as far as the people it worked for, they consider the issue fixed. If you don't get what I am saying, here it is. With such a poor install and test group for the "bug fix", chances are that all they are doing is causing one issue to go away while creating a totally different issue for other people. With something as complicated as audio and networking, I'll trust Microsoft over some dude and his roommate.Anonymous
August 29, 2007
George: 3rd party should be incapable of introducing glitches in Vista as long as MMCSS is running. With MMCSS present, there are basically only two things that can generate glitches. The first is long DPCs (which is the problem with the network), the second is if the disk is so utterly hammered that the I/Os to retrieve the multimedia data can't be read from the disk. I suspect that the glitches you saw may very well be a result of the same networking issue (but of course I don't know for sure since I've not collected perf traces on your machine). With the 10K throttleing you shouldn't see any glitches. Jumbo frames won't make a difference.Anonymous
August 29, 2007
Tim: Welcome to the world of Microsoft :). Both Raymond Chen and Ed Bott periodically goes into a rage about people ponying up advice on how to speed up Windows by tweaking registry keys that don't even exist. It's fun :).Anonymous
August 29, 2007
The comment has been removedAnonymous
August 29, 2007
Igor, I'm sorry you feel that way. It's possible that in the future, Mark will write a follow-on article which will express all the myriad of trade-offs involved in the discussions (which ranged from the capabilities of various network cards, processor design, scheduler design, test matrixes, and a boatload of other factors). But I doubt it. I mentioned the root cause of this issue above: The networking people weren't testing multimedia and the multimedia people didn't have the hardware necessary to test the network. Stuff happens, we realized the issue and we'll learn from it. One of the outcomes of the discussions about this problem is a better understanding of the issues that BOTH teams face to help ensure that mistakes like this don't happen again. And that is the last I will say on this subject. You can talk amongst yourselves if you like, but I'm done with this particular thread. Sorry about that.Anonymous
August 30, 2007
I'm a car guy and a computer guy and you are correct about your analogy; it isn't analogous... if a timing belt were to begin slipping, the engine, at the least, would quit running and would require physical attention before it would even run at idle speed again. At the worst, mechanical damage would occur, again requiring physical attention before running again. A better analogy, although still not perfect, is to say that as the engine has to run faster the ignition can't keep up with the higher speed requirements and occasional it mis-fires, causing a noticable "stutter". The faster the engine runs, the worse the mis-fires. Reduce the engine speed and the mis-fires go away.Anonymous
August 30, 2007
The comment has been removedAnonymous
August 30, 2007
"George: 3rd party should be incapable of introducing glitches in Vista as long as MMCSS is running. With MMCSS present, there are basically only two things that can generate glitches. The first is long DPCs (which is the problem with the network), the second is if the disk is so utterly hammered that the I/Os to retrieve the multimedia data can't be read from the disk. I suspect that the glitches you saw may very well be a result of the same networking issue (but of course I don't know for sure since I've not collected perf traces on your machine)." Oh we may have a miscommunications here. I was getting glitches in DVD playback while network test was HALTED and nothing else was going on. I was not having glitches while the 300 mbps test was in progress.Anonymous
August 30, 2007
The comment has been removedAnonymous
August 30, 2007
The comment has been removedAnonymous
August 31, 2007
Wilhelm Svenselius: Bugs happens, sure, but a Network throttling is not a BUG, its a design decision, it was designed to do that, its not there by a software bug, it was a intentionally introduced "feature". Then I ask, what kind of software designer comes with a such poor solution to the "Audio versus Network" problem?Anonymous
September 02, 2007
Igor and OSguy, I am a bit surprised by your comments, because the motherboard I'm using (Tyan K8WE) will physically disable certain devices (like the second NIC) if the second CPU socket isn't populated. Programming the APIC won't help, because physically the connection is gone. (OTOH, in my config, some IRQs will then be services by the second CPU, but NIC1 and my soundcard are still both on CPU0) It seems to me that it would be up to the CPU manufacturer to allow the second core to service interrupts, but it is none-to-obvious that this is the case, and it does not solve all cases (e.g. the typical multi-socket single-core configs). I'd love to see Larry address this in a blog posting, but the "Linux does!" type of argumentation strikes me as a bit pointless. (Linux can do many strange and not so wonderful things as well)Anonymous
September 02, 2007
The comment has been removedAnonymous
September 04, 2007
The comment has been removedAnonymous
September 04, 2007
The comment has been removedAnonymous
September 05, 2007
Igor is right, Vista distributes interrupts to both cores!Anonymous
September 06, 2007
Igor - The document you referred to talks about MSI and Interrupt Prioritization, both of which are different than dynamically routing interrupts to multiple CPUs based on load. The closest thing in the document which relates to interrupt routing is Interrupt Affinity - It clearly states that it is driver dependent to ask for such affinity - If driver does not ask specifically to route interrupts to say all CPUs in the system, the default machine policy takes over (and I suspect that is to route to only BP). ALSO of interest is that the documentation says it is a feature which may be useful for NUMA systems. Further it says - "It is important to realize that establishing an affinity policy represents a request by the driver and its device, not an absolute constraint." A decent scheduler does matter here - check out the Linux realtime preemption project where the scheduler offers to fix all latency sources that generate higher than ~1 msec latencies. Useful for this type of audio requirements. Google for Linux RT and Jackd.Anonymous
September 07, 2007
What's even worse is that this inherently flawed design impacts applications like Steam (a multiplayer gaming service that one normally leaves minimized in the system tray). Simply having Steam loaded into memory causes Vista to throttle the network connection.