Hyper-V and Multiprocessor VMs
Thanks for visiting our blog! I’m a development lead in the Windows Server Performance team and I led the performance effort on Hyper-V for Windows Server 2008 over the past three and a half years.
We’ve worked with the product team throughout the Hyper-V development cycle to deliver a competitive product and we’re excited about shipping Hyper-V RTM this year, with the Hyper-V Beta shipping in Windows Server 2008 this week!
Architectural Overview
Hyper-V uses a hypervisor-based architecture and leverages the driver model of Windows for broad hardware support. The hypervisor partitions a server into containers of CPU and memory. As a micro-kernel, it provides mechanisms for inter-partition communication upon which our new high-performance synthetic I/O architecture is built. The root partition owns physical I/O devices and provides services including I/O implemented by the virtualization stack to the child partitions.
The virtualization stack implements emulated I/O devices such as an IDE controller and a DEC 21140A network adapter. However, it is expensive to virtualize such devices. Sending a single I/O might require multiple trips between the virtualization stack and child partition. Instead, Hyper-V exposes synthetic I/O devices that are specially designed for VM environments. These devices are attached to VMBus, which is a plug-and-play capable bus that uses shared memory for efficient inter-partition communication. The Windows guests detect the devices on VMBus and loads the appropriate drivers.
Synthetic I/O in Hyper-V uses a client-server architecture with Virtualization Service Providers (VSPs) in the root and Virtualization Service Clients (VSCs) in the child. This architecture significantly reduces the cost of sending an I/O. Virtual Server customers should observe a major reduction in CPU usage in I/O-intensive loads when they migrate their VMs to Hyper-V.
In addition, we developed operating system enlightenments for Windows Server 2008, which make the NT kernel and memory manager smarter in VM environments, again to reduce the cost of virtualization.
Multi-Processor Guests
For this first blog post, I want to highlight one of the major performance features in Hyper-V: multi-processor virtual machines. Hyper-V supports 4P VMs for Windows Server 2008 guests and 2P VMs for Windows Server 2003 SP2 guests. For more intensive server workloads, you might consider virtualizing them in 2P or 4P VMs on Hyper-V. Of course, you should use multi-processor VMs only if the workload requires it since there is some cost to having additional processors.
However, operating system kernels and drivers use spin locks which do not block and spin until the lock is acquired, with the assumption that the lock is held for a short period. Virtualization breaks this assumption as virtual processors (VPs) are time-sliced. If a VP is preempted while holding a spin lock, other VPs may spin for a long time wasting CPU cycles.
We developed innovations in the hypervisor and Windows Server 2008 kernel to try to prevent long spin wait conditions and also to efficiently detect and handle them when they do occur. We also designed the hypervisor, including the scheduler and memory virtualization logic, to be lock-free on most critical paths to ensure good scalability on multi-processor systems.
As a result, Windows Server 2008 as a 4P guest scales well compared to the physical 4P system. This is one example of Windows Server 2008 as a guest and Hyper-V together providing performance advantages. We plan to continue to improving our scalability on multi-processor systems and multi-processor VMs in subsequent releases.
Closing Thoughts
Thanks for reading this far! I would encourage you to try Hyper-V Beta in Windows Server 2008, which launched this week. And take a look at the Windows Server 2008 and Virtualization web site for more information.
I look forward to writing more on our work on Hyper-V performance. Please add our blog to your RSS feeds!
Regards,
John Sheu
Senior Development Lead
Windows Server Performance Team
Comments
- Anonymous
January 01, 2003
What about memory sharing / overcommitted memory in hyper-v. How can I prevent from every VM to lock the MAX memory was assigned to it, while at the same time the VM does not using all the memory?
At the moment Hyper-V doesn't support memory sharing or overcommit. Tim Litton, Program Manager, Windows Server Performance.
Anonymous
April 04, 2008
Are there any advantages/disadvantages to configuring a Guest with more Virtual Processors then are physically present on the Host? (for example, a 4P Guest running on a 2P Host?) John: Each virtual processor needs to be scheduled on a physical processor. This means that the number of concurrently running virtual processors cannot exceed the number of physical processors in Hyper-V. Most operating systems are designed to expect that all of their processors are running at the same time, so having more virtual processors than physical processors may result in poor performance.Anonymous
May 08, 2008
How many VMs does hyper-v support? John: The number of active VMs that Hyper-V supports is only limited by the resources available on the host machine. For example, Hyper-V V1 supports upto 4 virtual processors per logical processor, so a host with 4 logical processors on it can support up to 16 virutal processors. These 16 virtual processors can be spread across the active VMs in a number of different configurations (such as 16 1VP VMs, or 8 2VP VMs, or 4 VP VMs, or some other mix). You should plan the capacity of the virtualization server based on its CPU, RAM, and I/O resources and the expected and peak loads on the VMs.Anonymous
June 25, 2008
The comment has been removedAnonymous
September 18, 2008
Are there any setting that can throttle CPU for the VMs so that overall host hyper-v cpu doesn't go to 100%?
Andy, yes it is possible to throttle the maximum CPU resources that an individual virtual machine gets. Under the settings for a virtual machine, select the “Processor” category and within the “Resource Control” section it is possible to set the limit on the amount of virtual CPU that can be allocated to this machine, as a percentage of the CPU resources available to this virtual machine. This limit setting applies regardless of whether or not other virtual machines running. If you want to ensure that the host CPU resources aren’t completely consumed, then you will need to ensure that sum of the limit settings of all the running virtual machines does not exceed 100%. However remember that server virtualization is about consolidating workloads of underutilized machines onto a smaller number of better utilized machines, so over committing on the virtual CPU is something that Hyper-V was designed to do. Tim Litton, Program Manager, Windows Server Performance.
Anonymous
October 01, 2008
Any tweaks for going over 4 cores? I'd like to virtualize a Ubuntu-based app that requires as many cores and as much memory as possible. -- At the moment Hyper-V only supports 4 virtual processors (VPs) per virtual machine. Tim Litton, Program Manager, Windows Server Performance.Anonymous
March 09, 2009
I've heard that Microsoft permits only 4 virtual machines per physical server - is this true? If not what's the maximum number of virtual machines permitted per physical machine -- Thanks for your question Vybez. The number of active virtual machines that Hyper-V can support is certainly higher than 4, and is dependent on the physical characteristics of the machine. Currently Windows Server 2008 Hyper-V supports up to 24 logical processors, with upto 8 virtual processors per logical processor. This means that a machine with 24 logical processors can run up to 192 virtual processors (24 * 8 = 192).
How the 192 virtual processors (VP) are used is quite flexible, it could be 192 VMs - each with 1 VP, or 96 VMs - each with 2 VPs, or 48 VMs - each with 4 VPs, or some combination in between. Tim Litton, Program Manager, Windows Server Performance.Anonymous
April 16, 2009
Is there a difference between Windows Server 2008 Enterprise and Standard with the amount of processors that the virtual machines can access? -- Thanks for your question Brad. When Windows Server 2008 is used as the guest operating system of a virtual machine, it can access up to 4 virtual processors. Standard Edition supports up to 4 processors, so both Enterprise and Standard editions will let the virtual machine use all 4 virtual processors. If your machine has more than 4 physical processors, you need to run Enterprise Edition of Windows Server 2008 for the root operating system to use all the processors to run the virtual processors. Tim Litton, Program Manager, Windows Server Performance.Anonymous
July 07, 2009
I have a dual quad core machine and I run windows 2008 server VM and configure 4 virtual processors to this VM. I have only one VM running on this hyper-v host, can this VM utilize the complete power of the dual quad core CPU?
Naveed - if the host is a dual quad core machine, it has 8 logical processors. If you only have one VM on the host, configured for 4 virtual processors, then Hyper-V will use up to 4 of the logical processors to run the workload of the VM. In this configuration, It is unlikely that all the logical processors on the host will be busy all the time. Tim Litton, Program Manager, Windows Server Performance.
Anonymous
July 13, 2009
Is there currently or will there be an option in Hyper V to use more than 4 virual processors per VM as we have an application which is cpu hungry we have 4x quad core processors but can only assign 4 logical processors per vm. At present, only 4 VPs are supported per VM in Hyper-V. Tim Litton, Program Manager, Windows Server Performance.Anonymous
August 11, 2009
Is there an improvement in performance in setting the Virtual Machine Reserve for each of my VM's rather than leaving that value at the default of 0? I have plenty of resources on the host but was wondering if performance is changed by specifically reserving processors for my VM's.
Specifying a Virtual Machine Reserve will ensure that a machine gets the specified CPU allocation - this can be very useful if you want to make sure that a particular VM is not CPU starved due to other VMs workloads. If you have plenty of CPU resource on the host to handle what you expect will be the peak load from the VMs (especially if you haven't overcommitted your logical processors), then setting a reserve is not likely change the performance for an individual VM. Tim Litton, Program Manager, Windows Server Performance.
- Anonymous
September 15, 2009
In the settings for a VM, is it possibile to change the "Percent of total system resources" in the processor resource control? I have a VM that is running too slow, because there is a process running that dosn't support multicore properly, so I need more power on only one core (like in Virtual Server 2005). -- Thanks for your question Jens. You've managed to stumble across one of the major differences between Virtual Server and Hyper-V. Virtual Server will only ever give one CPU to a virtual PC, so the maximum that you can allocate to any given virtual PC is 100 percent of one CPU.. With Hyper-V's support for multiprocessors it gets a little bit more complicated, as the amount of CPU resources that you are configuring will depend on the number of logical processors that are assigned to the virtual machine (VM). Here's an example, based on VM configured with 1 virtual processor on a quad-core machine (so it has 4 logical processors).
I've assigned this VM just one virtual processor, so when I set the "Virtual machine reserve" to 100%, the "Percentage of total system resources" that this VM can use is 25% (because I'm dedicating 100% of one of my four logical processors to this VM)
On Virtual Server, this is the same as setting Maximum Capacity to 100% for the virtual PC, because as noted above the maximum that you can allocate to any given virtual PC is 100 percent of one CPU. For information about Virtual Server and configuring CPU resources can be found at TechNet here:http://technet.microsoft.com/en-us/library/cc720296(WS.10).aspx. Tim Litton, Program Manager, Windows Server Performance.
Anonymous
November 12, 2009
My condition is the same as Naveed's and Jen's [see previous posts] -- (I have a dual quad core machine and I run windows 2008 server VM and configure 4 virtual processors to this VM. ) When you consider the contents of those posts, I understood that, if only one VM is running, its VM can not use full power of all physical CPU because, when only one VM is running, if affinity("virtual machine limit") is 100 then percentage of all CPU resource is 50%. Is this true? That’s correct. The system has a total of 8 logical processors. A single VM can have up to 4 virtual processors. Therefore, in this example, a single VM can utilize a maximum of 50% of that physical host’s CPUs. Tom Basham – Program Manager, Windows PerformanceAnonymous
December 07, 2009
I ran a test to compare physical Windows 2008 with 4 processors vs. one VM running Windows 2008 on Hyper-V with 4 virtual processors. I found the overhead to be a little bit more than 20%. I tried to optimize the Hyper-V according to Microsoft documentation available on TechNet. In TechNet, it is mentioned that Hyper-V CPU overhead is between 9-12%. Does a multiprocessor VM suffers more overhead? --- Yes, adding multiple virtual processors will increase the overhead (the exact amount will vary based on both the underlying hardware and the workload being executed). ThePerformance Tuning Guidelines for Windows Server 2008 R2 recommends: VMs that have loads that are not CPU intensive should be configured to use one virtual processor. This is because of the additional overhead that is associated with multiple virtual processors, such as additional synchronization costs in the guest operating system. More CPU-intensive loads should be placed in 2-VP or 4-VP VMs if the VM requires more than one CPU of processing under peak load. Tom Basham, Program Manager, WIndows PerformanceAnonymous
January 24, 2010
I've noticed that the Hyper-V settings will allow you to assign 4 processors to a Windows 2003 Server virtual machine. I understand that only 2 are supported. We have tried 4 in a test enviroment and the virtual machine seems to work fine. What are the potential consequences of this configuration? It's an unsupported scenario, but not explicitly blocked. Windows Server 03 will only utilize 2 processors in this case. There's potential for performance degradation using more processors. Tom Basham, Virtualization Performance PM, Windows Fundamentals TeamAnonymous
September 21, 2010
If the guest OS is the Windows 2003 R2 ENT edition, up to how many CPUs can the HyperV "provide" to the vm ? Hi Christos, Hyper-V supports up to 2 virtual processors for a Windows 2003 R2 ENT guest OS. For a complete listing of the supported number of virtual processors for other guest OS, please seehttp://technet.microsoft.com/en-us/library/cc794868(WS.10).aspx. Thanks.Anonymous
November 30, 2010
The comment has been removedAnonymous
December 07, 2010
If I may ask a supplementary question, if I have a single threaded CPU bound application (i.e. a scientific workload) and that is the only work I wish to run on that VM then, as I understand the previous answer, creating a VM with 4VPs will result in 4 logical processors being scheduled to the VM each and every time it is dispatched by Hyper-V. So even though my application can only use one VP at a time (dispatched by Windows across all 4VPs) most of the time (Windows overheads and interrupts for this type of workload are small), Hyper-V will still schedule 4VP. Does this mean that approximately 3 logical processors are effectively wasted, because no other VM can get access to these LPs while the application is running on this VM and that I would be better off configuring the VM with 1 or 2 LPs?
If you are running a single threaded application, then you are better off configuring the VM with 1 or 2 virtual processors (VPs). More information about configuring the VM in terms of VP count and monitoring performance can be found onhttp://www.microsoft.com/whdc/system/sysperf/Perf_tun_srv-R2.mspx.
The Tuning Guide covers things like how VPs are scheduled on the system's logical processors (LPs) and other topics you might find useful for your use case.