Azure A-SERIES, D-SERIES and G-SERIES: Consistent Performances and Size Change Considerations
LAST UPDATED: November 11th, 2015.
Thank you for visiting this page, before going forward, let me clarify immediately which are the two key messages I want to give you with this blog post:
- Do not waste your time doing CPU tests and comparisons: if you ask Azure A-SERIES VM, you will simply obtain it and performances will be consistent independently from the Azure hosts underlying CPU architecture.
- Before choosing your VM SKU and size consider your future scaling requirements: depending on how you choose your VMs, and the order you allocate, future downsizing or upsizing may require VM export/import procedure and cannot be done “in-place”.
Regarding the first point, even if I will show you some CPU details and performance tests results, I’m *not* recommending you to do that yourself simply because it is unuseful: Azure does an excellent job providing consistent performances for A-SERIES VMs, you don’t need to worry about where your VM is allocated, you will get what you ask and pay.
Introduction
When Microsoft originally released IaaS Virtual Machines (VM) with A-SERIES SKU, there was not official mention on the CPU type used. You can find over the Internet tons for blog posts and other content types debating this fact, you can realize that creating VMs yourself and look at what Windows Server Task Manager or Resource Monitor, but again there is no official mention in the master page below:
Virtual Machine and Cloud Service Sizes for Azure
http://msdn.microsoft.com/en-us/library/azure/dn197896.aspx
Something then changed with the introduction of compute intensive HPC instances, that is A8 and A9 VM sizes powered by a second NIC with Infiniband capability. For the first time, in the article mentioned above, you can find a specific mention to INTEL based CPUs:
About the A8 and A9 Compute Intensive Instances
http://msdn.microsoft.com/library/azure/dn689095.asp
Back in March 2015, Microsoft also announced new A10 and A11 VM SKUs, with an interesting characteristic: if you need the same Intel Xeon E5-2670 CPU power as A8/A9, but you don’t need the second Infiniband/RDMA NIC for MPI applications, now you
have an effective option to cut down cost of these normally expensive VM sizes.
New A10/A11 Azure Compute Sizes
http://azure.microsoft.com/blog/2015/03/05/new-a10a11-azure-compute-sizes
Last year, Microsoft introduced a new VM SKU called D-SERIES, but also this time CPU specs were not officially specified, the original official blog post simply talked about 60% faster CPU compared to A-SERIES. You will not able to find this confirmation in neither the first article mentioned above nor the blog post below announcing new SKU release:
New D-Series Virtual Machine Sizes
http://azure.microsoft.com/blog/2014/09/22/new-d-series-virtual-machine-sizes
Finally, Microsoft recently released the new G-SERIES VM SKU equipped with the latest Intel® Xeon® processor E5 v3 family, able to provide up to 32 cores (no hyper-threading!), up to 448GB RAM and up to 6.5TB local SSD drive. At the time of writing this blog post, this is the biggest and most powerful VM available in the Cloud!
Largest VM in the Cloud
http://azure.microsoft.com/blog/2015/01/08/largest-vm-in-the-cloud
Now, let me recap which CPU types you have on all the various VM SKUs I mentioned above and something else that will come in the next future:
- A-SERIES A0-A7: not officially specified
- A-SERIES A8-A9-A10-A11: Intel® Xeon® E5-2670 2.60 Ghz
- D-SERIES: not officially specified
- DS-SERIES: not officially specified
- G-SERIES: Intel® Xeon® processor E5 v3 family
NOTE: G-SERIES will be able to mount up to 64 disks with STANDARD_G5.
Regarding DS-SERIES, it has been announced at TechEd Europe and its purpose it to leverage the future “Azure Premium” Storage feature, specifically the ability to provide >50K IOPS. More details at the link below:
New Premium Storage-Backed Virtual Machines
http://azure.microsoft.com/blog/2014/12/11/new-premium-storage-backed-virtual-machines
Changing VM size
IMPORTANT: What is reported in this section only applies to VM (v1) deployed with Azure Service Management API (ASM) and *does* not include VM (v2) deployed using new Azure Resource Manager (ARM) API. For considerations on VM SKU/size changes in ARM, please read my blog post below:
Azure VM Resizing and SKU Change in ASM and ARM
Since the introduction of D-SERIES SKU, there was the misconception that A-SERIES VMs (except A8/A9/A10/A11) used a specific CPU type while D-SERIES VMs used a different one: the short story is that this is not true. Let me emphasize again that, except for A8/A9/A10/A11, Microsoft did not release any official specification on the CPU type used for A-SERIES and D-SERIES. The key concepts to keep in mind here are that Azure datacenters are composed by clusters of uniform hardware servers and that a single Cloud Service cannot span an Azure cluster. Now, what happen if you create a new VM in a new Cloud Service with A1-A7 size? In which cluster that VM will be allocated? What I found during my recent tests is that you may land on Azure cluster with different capabilities and you will not have any control on choosing which one. In the print screens below I reported two different trials where it happened that my A1 VM landed in two different Azure cluster types:
In the left case, you will not be able to upgrade to D-SERIES SKU in-place, you will have to export disks and create a new VM using the previously saved disks. This may be a problem if in the future you will want to upgrade the VM type/size and/or join different VM types/sizes to the same Cloud Service. You can also realize that for A8/A9 HPC instances, there is a special allocation strategy, see last section on A8/A9 instances in this blog post for more details. If you try to update the VM to D-SERIES (or A8/A9) using PowerShell Set-AzureVMSize commandlet, you will obtain this error message:
Update-AzureVM : Compute.CannotUpgradeDeploymentToNewRoleSize : Unable to upgrade the deployment. The requested VM size 'A9' may not be available in the resources supporting the existing deployment. Please try again later, try with a different VM size or smaller number of role instances, or create a deployment under an empty hosted service with a new affinity group or no affinity group binding.
This finding generate two practical questions:
- How can I be sure to allocate my A0-A7 VMs in a cluster that will guarantee future in-place upgrades?
- Performances of the same A0-A7 VM will be equal in both cases?
Regarding the first question, I found a simple workaround: when you create your very first VM in your Cloud Service, be sure to specify one of the D-SERIES size even if you do not need it immediately. Doing this, your Cloud Service will be “tied” to a cluster that will support both A-SERIES (except A8/A9) and D-SERIES, then for all the future VMs contained in the same Cloud Service. Now, you can create additional A-SERIES VMs and mix together in the same Cloud Service. If you do not need the first D-SERIES VM, you can now safely delete it. If you need to create multiple Cloud Services, you can use a slightly different approach: create an “Affinity Group” and specify it when creating the very first VM in the very first Cloud Service, then you can reuse the “Affinity Group” to create more objects. Be aware that “Affinity Group” usage is not recommended anymore and maybe discontinued in the future, but at the moment it is still available and can be used.
Regarding the second question, it is important to provide an answer since there is a cost difference between A-SERIES and D-SERIES for the same VM size: you do not want to pay an A1-A7 VM and eventually have lower performances. Here Azure provides a perfect answer as I can demonstrate you below: even if your A1 (for example) VM may be allocated on different cluster types, compute power provided will be same. Here I can only guess that the Azure Hypervisor is able to limit the CPU power for A1-A7 VMs when running on newer most powerful clusters. I used several CPU benchmarking tools, the results were pretty consistent and show almost the same performances in both cases. Be aware that there is also a small 5-10% variation running the same test at different times.
IMPORTANT: Details and specs for CPU may change in the future without notice. Results shown here are only for demonstration purposes and may change as well..
Again, let me emphasize: do not waste your time doing CPU tests and comparisons. If you ask Azure A-SERIES VM, you will simply obtain it and performances will be consistent independently from the Azure hosts underlying CPU architecture.
A8/A9/A10/A11 HPC instances VMs
Even if these four VM sizes are parts of the A-SERIES SKU, they have a special treatment. Let’s create an A9 VM in West Europe region with the following Power Shell script:
Now, I’m going to try downgrading to A5 instance size using the command below:
Surprisingly, it will fail! If you think about the peculiarity of A9 (and A8), that is Infiniband NIC capability, you can easily guess that this VM has been allocated to an Azure cluster with dedicated and specialized hardware: since the resize operation cannot change the Azure cluster allocation, the final conclusion is that you will not be able to change the VM size, except for A8 size that falls in the same HPC category.
After creating a new Cloud Service containing A8/A9, you will not able to deploy other VM sizes in the same Cloud Service except for the same A8/A9. The explanation is still the same: since HPC instances use dedicated Azure clusters with specialized hardware, and since a Cloud Service cannot span different/multiple Azure clusters, you will not able to deploy different VM sizes other than HPC instance sizes. The only workaround here is to create a new/different Cloud Service and join it to the same Azure Virtual Network.
The only way to change the VM size without losing any data is to export the VM definition, delete the VM preserving the VHDs, then import back the definition after applying the changes you need. I’m going to provide you an example of this procedure in the next section. Regarding the latest A10/A11 HPC instances, since they do not have a second Infiniband NIC, but still the same powerful Intel Xeon CPU, you may try to guess that another different type of Azure hosts would be used. Surprisingly, it is not true, you can mix A10/A11 VMs in the same Cloud Service with A8/A9, then the same concepts apply to A10/A11 VMs as described in this section for A8/A9.
Additionally, you can downgrade or upgrade between all A8, A9, A10 and A11 VM sizes.
G-SERIES VMs
For the latest G-SERIES VM SKU, the situation is pretty similar to what I described for HPC instances in the previous section. If you create a new VM and chose a G-SERIES SKU size, your VM will be created on an Azure cluster type that will support only G-SERIES SKU and sizes, then if you want to downgrade to a different SKU (A-SERIES, D/DS-SERIES), you will need to export the VM as outlined in the next section. Changing VM size inside the same G-SERIES SKU is allowed:
Additionally, if you want to create a new G-SERIES VM and join an existing Cloud Service, the operation will fail if the Cloud Service contains VM SKUs different from G-SERIES (smaller SKUs). The reason is simple: your Cloud Service cannot span multiple Azure clusters, since it is already allocated on a cluster type not supporting G-SERIES, then the operation will fail.
Then, if your Azure IaaS environment must contain G-SERIES and other VM SKUs (A-SERIES, D/DS-SERIES), the only solution is to create a separate Cloud Service for G-.SERIES VMs and eventually join the same Virtual Network. Be aware that you will need to use the latest/newest “Regional” Virtual Network (default) because “Local” Virtual Network based on old concept of Affinity Group will tie your VMs to the same and unique Azure compute cluster, then raising the same kind of allocation problems just described here.
Regional Virtual Networks
http://azure.microsoft.com/blog/2014/05/14/regional-virtual-networks
VM Export and Import
IMPORTANT: What I'm going to describe in this section is necessary only for VM v1 (ASM). In VM v2 (ARM), dynamic resize is possible without exporting/importing.
There are various ways to migrate a VM preserving existing OS and data disk, in my personal opinion to simpler and best to use is based on Power Shell “Export-AzureVM” and “Import-AzureVM” commandlets. Attached to this post there is a test script I used to validate a specific scenario: feel free to re-use but keep in mind that it is only a sample to demonstrate the functionalities and not for production purposes. It is related to a specific scenario with specific requirements, then you should carefully test in your lab environment to verify the results. The general workflow is based on the following steps:
- Export the VM configuration with “Export-AzureVM” Power Shell commandlet: an XML file will be generated on your local machine.
- Manually edit the generated XML file to change the VM size: this file will contains all disks and virtual network configuration information:
- Remove the original Cloud Service and VM.
- Create a new Cloud Service, with the same name, and a dummy VM of D-SERIES size: this will force Azure to allocate future VMs, for the same Cloud Service, on a cluster able to support this SKU and A-SERIES as well.
- Import back the modified XML configuration file with “Import-AzureVM” Power Shell commandlet.
That's all folks! I hope you will finds this content useful, your comments are welcome!
As usual, you can also follow me on Twitter at @igorpag. Regards.
Comments
Anonymous
January 15, 2015
This is terrible ! Azure should take care of this! This means terrible downtime and headache which cannot be the goal of Azure.Anonymous
April 30, 2015
Very useful, Thank youAnonymous
June 03, 2015
The comment has been removedAnonymous
July 08, 2015
It helps a lot, Thank youAnonymous
November 03, 2015
Creative suggestions . I Appreciate the info . Does anyone know if my business would be able to get a sample 2013 DS-11 form to use ?Anonymous
November 03, 2015
Hi Onie,not sure to understand what you mean with 2013 DS-11....Anonymous
January 21, 2016
Hi Igor, Very nice article. I was recently trying to run some tests to check how much work load (in term of parallel compute-intensive requests) a D5v2 (16 Core) can perform vs. say a D4v2 (8 Core) VM. For some strange reason I found that D5v2 was much worse than D4v2. However I am not able to repeat those results now though I was able to do it few days ago. So my question is: Is there a performance guarantee for a given number of virtual cores or does this vary depending on how "loaded" or "utilized" the hardware pool is? If its the former then is this repeatable over long periods of time (say 6 months). If its the latter then its very difficult to gauge the expected work throughput and will vary wildly depending on time of day or day of week or number/load of other VMs. It would be good to hear some thoughts from you on this. My final goal is to actually estimate the infrastructure costs for serving a given set of users (at a defined, acceptable level of performance). These numbers would also help in planning out the scaling as number of users grow. Thanks!Anonymous
January 22, 2016
Hi Santosh, Be aware that Azure does NOT overcommit memory/cores as other Cloud Providers: if you buy a core, it is physical and totally allocated to you (except S0 that is a slice of 1 core), same for memory and same for Premium Storage. I don't know the details of your performance tests, but I'm suspecting something strange in the way you tested. Your performance test is only CPU/Memory bound or storage and networking are also important? Did you use Premium Storage in case storage is important? Regards.Anonymous
January 22, 2016
.... additionally, Azure does not use HyperThreading as other Cloud Vendors.Anonymous
February 23, 2016
Igor, Thoughts on this post: www.pythian.com/.../comparing-cpu-throughput-of-azure-and-aws-ec2 ?Anonymous
February 24, 2016
Yes, you can forget it :-) Both Microsoft and Amazon constantly update CPUs and infrastructure at a such velocity that a performance test 1-year old is simply no longer valid. Regards.