Increase VMBus buffer sizes to increase network throughput to guest VMs

Under load, the default buffer size used the by the virtual switch may provide inadequate buffer and result in packet loss. We recommend increasing the VM bus receive buffer from 1Mb to 2Mb.

Traffic jams happen every day, all across the world. Too many vehicles competing for the same stretch of road, gated by flow control devices like stop signs and traffic lights, conspire to ensnare drivers in a vicious web of metal and plastic and cell phones. In the technology world, networking traffic is notoriously plagued by traffic jams, resulting in all sorts of havoc, including delayed web pages, slow email downloads, robotic VOIP and choppy YouTube videos. (Oh, the humanity!)

Virtualized networking can be complicated, what with the root and child partitions relaying packets across the VM bus to reach the physical NIC. The VM bus, anticipating contention, uses buffers to queue data while the recipient VM is swapped out or otherwise not keeping up with the traffic. The default buffer size for WS08 R2 is 1Mb, which provides 655 packet buffers (1,600 bytes per buffer).

The hypervisor, meanwhile, calculates a scheduling interval, or quantum, derived from the system's interrupt rate. The hypervisor attempts to ensure every VM has a chance to run within that interval, at which time the VM wakes up and does whatever processing it needs to do (including reading packets from the VM bus). At very low interrupt rates, that quantum can be nearly 10ms.

Whereas the native system handles on the order of 260,000 packets/second, virtualized systems, in some scenarios, can—in the worst case scenario—begin seeing packet loss under traffic loads as low as 65,500 packets/second. This isn’t an inherent tax incurred by virtualizing or a design limit; rather, it’s the result of specific characteristics of server load requiring more VM bus buffer capacity. If the logical processors hosting the guest partitions are receiving very few hardware interrupts, then scheduling quantum grows larger, approaching 10ms. The longer scheduling quantum results in longer idle periods between VM execution slices. If the VM is going to spend almost 10ms asleep, then the VM bus’ packet buffers must be able to hold 10ms worth of data. As the idle time for a VM approaches 10ms, the maximum sustainable networking speed can be calculated as:

655 default packet buffers / ~10ms idle interval = maximum 65,500 packets / second

We can increase throughput, though, by increasing the amount of memory allocated to the buffers. How much should it be increased? On paper, 4Mb is the maximum useful size; a 4Mb buffer provides about 2600 buffers, which can handle 10ms’ worth of data flowing at approximately 260,000 packets per second (the max rate sustainable by native systems). In reality, depending on the workload, the VM’s swapped-out time probably doesn’t approach the maximum 10ms quantum. Therefore, depending on how frugal you want to be with memory, increasing to 2 Mb is probably adequate for most scenarios. If you’re living large in the land of RAM, lighting your cigars by burning 4Gb memory sticks, then go for broke, cranking the buffers up to 4Mb.

The buffers are allocated from the guest partition’s memory and updating the buffer size requires, per each guest VM, adding two registry keys. To increase the buffer size, we first need the GUID and index associated with the network adapter. In the guest VM, open the Device Manager, expand Network Adapters , right click Microsoft Virtual Machine Bus Network Adapter and choose Properties (if you have an a driver marked “(emulated)”, you should take a detour to install Integration Services from the VM’s Action menu, then add a new synthetic network driver through the VM setup. See https://technet.microsoft.com/en-us/library/cc732470(WS.10).aspx , step 3 for instructions).

On the Network Adapter Properties dialog, select the Details tab. Select Driver Key in the Property pull-down menu as shown in figure 1 (click the images to see a version that's actually readable):

Record the GUID\index found in the Value box, as shown in figure 1, above. Open regedit and navigate to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\ {GUID}\{index} as shown in figure 2:

Right click the index number and create two new DWORD values, entitled ReceiveBufferSize and SendBufferSize (see figure 3). These values measure the memory allocated to buffers in 1Kb units. So, 0x400 equates to 1,024Kb buffer space (the default, 640 buffers). In this example, we’ve doubled the buffer size to 0x800, or 2,048Kb of memory, as shown in figure 3:

Your workloads and networking traffic may not need increased buffers; however, these days, 4Mb of RAM isn’t a tremendous amount of memory to invest as an insurance policy against packet loss. Now, if only I could increase a few buffers and alleviate congestion on my daily commute!

Tom Basham

Virtualization Performance PM, Windows Fundamentals Team