Hyper-V CPU Scheduling–Part 1
The other day I was approached by someone at Microsoft with questions about how Hyper-V schedules virtual processors, and how customers use the controls we provide. I was surprised to discover that I had not blogged on this topic.
Today I want to (begin to) fix that oversight.
Before I get started – there is one concept that I need to get straight. That is the concept of “contention for resource”. Most of the aspects of the Hyper-V CPU scheduler only come into play when all of the CPU resource on a physical computer is being used and Hyper-V has to decide who wins and who loses. This situation (all the CPU being used, and virtual machines wanting more) is where we say there is “contention for resource”. If there is free processor resource on the physical computer – there is no contention and (some) things behave differently.
Next, because it can get a bit complicated to talk about how CPU scheduling works with virtual machines and physical computers, I am going to try and use examples as often as possible. But to keep it simple I am going to use the same fictitious computer for all examples – which is a single processor quad core system (so a computer with 4 physical cores). And, unless I specify otherwise – all virtual machines will have 4 virtual processors.
With that out of the way – lets talk about the things you can configure:
Virtual machine reserve
This is configured as a percentage of the total possible CPU usage of a virtual machine. The virtual machine reserve functions in a couple of ways.
The first one is when a virtual machine is turned on. We will not let a virtual machine turn on if we determine that it is not possible to honor the virtual machine reserve. For example, if you configured each of your (4 virtual processor) virtual machines (on our fictitious computer) with a reserve of 20% we would only let you start 5 virtual machines. When you tried to start a 6th virtual machine you would get an error message stating that you cannot initialize the virtual machine.
Note that you will get this error even if all 5 running virtual machines are completely idle and are not using any CPU resource.
The second way that the virtual machine reserve is used is when there is contention for resource on the CPU. Simply put – if there is higher demand for CPU than is physically available, we will ensure that a virtual machine that needs CPU resource gets at least its CPU reserve.
What surprises some people is that we do not enforce the reserve when there is no contention for CPU resource. To explain what I mean by this:
Imaging our fictitious setup with 5 virtual machines, each with a 20% reserve, and all 5 virtual machines running and idle (0 CPU utilization). At this point in time all CPU resource has been reserved. Now, what will happen if one of these virtual machines tries to use 100% of available CPU resource while the others remain idle? In short – we will let the virtual machine have 100% of the available CPU.
The reason for this is that we can always take CPU resource back from a virtual machine. So as long as we know that the sum total of running virtual machines do not exceed 100% CPU reserve, we can let any given virtual machine chew into another virtual machines reserve, because when it is needed we can make sure the reserve is honored.
A final interesting note about the virtual machine reserve is that by default it is set to “0%” – which means the reserve is disabled. Furthermore – if you have a configuration like the one we have been discussing where all CPU is reserved, you can still start extra virtual machines as long their CPU reserve is set to 0%.
The lesson to learn here is that you should not get over-zealous with the virtual machine reserve, as things can get confusing.
How do people use the CPU reserve?
There are a couple of ways that the CPU reserve gets used:
To enforce SLAs
This is the most common use that I hear of the CPU reserve – to be able to provide a guarantee about the minimum level of CPU that will be available for a virtual machine.
One interesting note to make about this is that most of the time the CPU reserve is used by the Hyper-V administrator to make someone else happy. Put simply – it provides a way for the Hyper-V administrator to assuage the fears / concerns of a user or client who does not have a high level of confidence in virtualization yet.
To stop people from running too many virtual machines
While it is not really what it was designed for – many people use the CPU reserve as a way to ensure that they do not accidentally run more virtual machines on a given server than they want to. For example: if you default to a CPU reserve of 20% – you can never have more than 5 virtual processors on each physical core.
What are the common problems involved in using a CPU reserve?
The most common issues associated with using CPU reserves are:
Lack of flexibility
As I have mentioned, having CPU reserves configured can stop a virtual machine from being started if the reserve cannot be met. So if you have a system setup “just right” with 10 virtual machines – each with a 10% reserve – what do you do when you want to start one more? Do you setup the 11th virtual machine with no reserve? Or do you go and change the reserve on all the other virtual machines to 9%?
There is a similar consideration when using CPU reserves in a cluster – if they are configured poorly – a virtual machine may fail to start when it is failed over, because the total reserve has been exceeded.
The bin-packing problem
There is an interesting problem that can happen once you start playing with the CPU reserve. The bin-packing problem.
For example: on our fictitious quad-core system imagine that you have two virtual machines. One is a 4 virtual processor virtual machine with a 50% reserve – which is to say that it gets half the physical computer reserved for it. The second virtual machine is a 2 virtual processor virtual machine with a 80% reserve – which is only 40% of physical computer.
You may be thinking that 50% plus 40% is under 100% (90%) so everything will work fine, but it does not.
The first virtual machine will start with no problem, but when you go to start the second virtual machine we will try to find two physical processor cores that each have 80% unreserved. But there are none, as all 4 cores only have 50% unreserved.
That is pretty much all that I have to say about the CPU reserve. Tune in tomorrow – and I will continue by discussing the “virtual machine limit”
Cheers,
Ben
Comments
Anonymous
February 16, 2011
Hello! Do things get complicated when hyper-threading is enabled? Example: Core i-530, 2 physical cores, hyper-threading is on, so 4 logical processors. Hyper-V is running 2 virtual machines with 2 virtual processors each. If both virtual machine's reserve is set to 50 does it reserves all CPU's capacity (and 3rd VM with reserve>0 will not start) or just the half? Btw, is hyper-threading turned on brings any goodness for Hyper-V? The host is not used for other purposes, one VM is corporate proxy/VPN Windows 2003 Server serving not very large Internet bandwidth 2MBit/s and the other is virtualized desktop Windows XP.Anonymous
February 18, 2011
Hello, if I have 2 Xeon(R) CPU E5620 processors(8 cores) and have 8 VMs, how do I assign the cpus? Currently, I give each VM 4 cores.Anonymous
February 18, 2011
IL - At this stage you will only have half the system reserved (2 * 2 * .5 == 2 processors reserved). Hyper-threading may not always be beneficial - but it will not cause harm, so I would leave it on. Cloudtech - Hyper-V does not allow you to manually assign specific processors to specific virtual machines. Cheers, BenAnonymous
February 21, 2011
My experience is that if you have a host with multiple processors, you have an artificial limit on guest processing consumption. For example, with a quad quad-core configuration, the most any one VM can consume is 25%--if it is assigned four virtual processors--even if all the other VMs are idle.Anonymous
September 12, 2011
The comment has been removedAnonymous
November 30, 2011
How come these settings are not available in SCVMM 2012 on a VM? SCVMM GUI should be a superset, not a subset, of the GUI?Anonymous
November 04, 2012
Hi Ben, Have a follow up question...usually processors have multipliers as 400 * 6 = 2.4 GHz. How does this value map to cores and CPUs. Does each core perform at 2.4 GHz or do all the CPU Cores crunch instructions at 2.4GHz collectively?