My personal Azure FAQ on Azure Networking SLAs, bandwidth, latency, performance, SLB, DNS, DMZ, VNET, IPv6 and much more
Last updated: May 17, 2015 (including BUILD and IGNITE announcements).
IMPORTANT: The information I reported in this post is time sensitive since may change in the future. I will try to maintain the content updated, but you should double--check with official Azure documentation sources.
In the last three years I worked extensively on Azure IaaS, during engagements with my partners I have been asked many questions on Azure networking in general, and more specifically on Virtual Network (VNET) and VMs. Providing adequate answers is not always easy, sometime the documentation is not clear or missing in details, while other times you can deduct yourselves but requires pretty good knowledge. Since I detected some recurring patterns in these questions, I decided to write my personal FAQ list in this blog post. Just to be clear: even if I’m a MSFT and have inner knowledge of Azure, I’m not going to unveil you any reserved or secret information. Everything you will read here can be found playing directly with Azure or retrieved by using official and public documentation. If you have additional “nice” questions requiring a non-trivial or non-easy-to-retrieve answer, fill free to insert a comment below this post. Additionally, if you have proposals or feedbacks regarding new feature that Azure networking should have, please use the link below to submit your ideas or vote existing ones:
FEEDBACK FORUM: Networking (DNS, Traffic Manager, VPN, VNET)
As usual, you can also follow me on Twitter at @igorpag. Regards.
Can I use Receive Side Scaling (RSS) optimization in Azure VM?
YES. Azure HyperV is able to support RSS on the host side, but to leverage this functionality in Guest OS you need to manually enable it following the instructions at the link below:
Virtual Receive-side Scaling in Windows Server 2012 R2
https://technet.microsoft.com/en-us/library/dn383582.aspx
For Windows Server, Guest OS must be 2012 R2 version or later, while for support on Linux OS, see the article below:
Linux and FreeBSD Virtual Machines on Hyper-V
https://technet.microsoft.com/en-us/library/dn531030.aspx
Azure “Affinity Group” is officially deprecated?
YES. The introduction of Azure Resource Manager (ARM) new API model, “Affinity Group” (AG) is officially deprecated, thus not recommended to use it.
Azure Subscription and Service Limits, Quotas, and Constraints
http://azure.microsoft.com/en-us/documentation/articles/azure-subscription-service-limits
Even if AG can be still used in Service Management (SM) API, but not in the legacy and new Azure portals, usage is not recommended since many Azure networking features rely on “Regional Virtual Networks”. Using an AG to pin a Virtual Network in fact, will originate a “Local Network” tied to a single Azure Cluster and will not have benefits of the new Azure scale units for compute resources.
Does ICMP work in Azure?
It depends from where you want to use ICMP tools and which targets you want to reach. ICMP works inside an Azure Virtual Network, between different Azure Virtual Networks connected via VPN or ExpressRoute or hybrid connections between on-premise and Azure VNET. It will *not* work if you have to cross the Azure Load Balancer (SLB), for example if you want to ping a Cloud Service from another Cloud Service. If you need ICMP for diagnostic and troubleshooting, there are valid alternatives to PING and TRACERT not based on ICMP but TCP. My favorite ones, among many others available in Internet, are PSPING from Mark Russinovich SysInternals tools and NMAP:
PsPing v2.01
https://technet.microsoft.com/en-us/sysinternals/jj729731.aspx
NMAP
Does Azure provide SLA on VM network bandwidth?
I tested VM bandwidth limits on almost all VM sizes and found the results pretty consistent over time. Since official numbers have not been disclosed yet, I will not share my findings. You probably found in Internet, some old tables and information related to maximum network bandwidth for various VM SKUs, you should not trust them since outdated and Azure infrastructure evolved in the meanwhile. You can still use that info to have a general idea, but you will need to wait for the official release of associated SLA.
Does Azure provide SLA on VM-to-VM network latency?
During my tests and projects, Azure network Always provided excellent performances in terms of latency. At least at my knowledge, there is no Cloud vendor able to guarantee a maximum network latency, backed up by a formal SLA, between two different VMs. It’s very easy to test yourself and have a good understanding of average latencies, but you need to be very careful on your test criteria. First of all, you need to test over various hours during the day including peak and non-peak hours, various days over the week including weekends and working days, then finally you should use calculate the average and consider a good percentile sampling near the 95% to eliminate peaks and occasional strange values. Additionally, don’t assume latencies will be the same in all Azure datacenters. If you want to do your own tests, I would recommend you PSPING tool from SYSINTERNAL suite that you can find here: http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx. What is really nice regarding this tool, is that it will not use ICMP then no problem to traverse firewalls and load-balancers, you can decide which TCP port to use and, very important, you can test for bandwidth, not only for latency. You can read full details on using this tool to test Azure latency in one of my previous blog post:
Azure Network Latency & SQL Server Optimization
In the past, “Affinity Group” concept has been used to provide co-location of VMs and minimize latency: now this concept is officially deprecated with the new Azure Resource Manager (ARM) API and broad adoption of “Regional” VNET. Azure infrastructure continuous evolutions made network more efficient and performant, then you should not use it anymore.
Azure Datacenter-to-Datacenter traffic will go through public Internet?
NO. Traffic between VNETs in different datacenters, connected via VPN connection will remain inside Microsoft owned network backbones, see slide [34] in the presentation below:
Introduction to Microsoft Azure Networking Technologies and What's New
https://channel9.msdn.com/Events/TechEd/Europe/2014/CDP-B227
Additionally, in the case of VNETs connected using Express Route, traffic will not go through public Internet, private leased line will be used.
Is it possible to white-list Azure public IP addresses?
YES, all Azure public IP ranges are published at the link below. If you want to white-list, for security reasons, on your on-premise network you can do that selectively by region but not by service.
Microsoft Azure Datacenter IP Ranges
http://www.microsoft.com/en-us/download/details.aspx?id=41653
If you rely on specific Azure resources, for example an Azure SQLDB instance or blob storage account, be aware that the assigned IP you will see, resolving their specific DNS names, may change without notice. IMPORTANT: don’t rely on any geo-location service for Internet IP used by Azure since may report incorrect information, for more details see the blog post below:
Microsoft Azure’s use of non-US IPv4 address space in US regions
If you created your own Cloud Service, and want to be possible to validate it using reverse-DNS, you can do that enabling PTR record registration (by Azure) for your Cloud Service VIP as described at the link below:
Announcing: Reverse DNS for Azure Cloud Services
http://azure.microsoft.com/blog/2014/07/21/announcing-reverse-dns-for-azure-cloud-services
Be aware that the Virtual IP (VIP) associated to your Cloud Service may change under some conditions, if you need to be absolutely sure it will never happen, you need to enable "Reserved IP" feature as described at the blog post below:
Reserved IP addresses for Cloud Services & Virtual Machines
http://azure.microsoft.com/blog/2014/05/14/reserved-ip-addresses
What is “*.internal.cloudapp.net” DNS suffix in Azure VM?
If you build an Azure VM and join to an Azure Virtual Network and try to ping it using the VM host name (legacy Netbios name to be clear), this is what you will see:
As you can see in the picture, there are two strange parts in the FQDN: nothing strange in “cloudapp.net”, but what are “internal” and “a3”? You can easily guess that “internal” is the default DNS sub-zone that Azure internal DNS (iDNS) will use to host records for VMs and resolve to internal VM IP (DIP). “a3” instead is a little bit complex: in short, it’s describing the network zone, inside the specific Azure datacenter, where the VM is allocated. There is no official documentation or list of these zones, then don’t ask more, but please note that
it will change not only between Azure datacenters, but also inside a single datacenter.
Is there any latency overhead in connecting VMs in different Cloud Services?
The answer is NO, or at least it is negligible. In the recent past, I worked with several partners concerned about placing the application VMs in one Cloud Service and then placing the backend VMs (SQL Server) in a different Cloud Service. This is the typical scenario if you want to use SQL Server AlwaysOn Availability Group (AG) mechanism. Then, even if the application VM will connect to the backend VM through the Azure Load Balancer (SLB), the network latency overhead is really minimal due to an Interesting network optimization in Azure. During the initial TCP connection handshake between the two VMs, Azure will recognize that the communication is between two internal resources and will allow direct communication as in the case of network connection using DIP inside the same Cloud Service. You can read more about measurements I have done in the past in my blog post below:
Azure Network Latency & SQL Server Optimization
Ping and Tracert tools will work inside Azure Virtual Network?
YES, they will work perfectly inside a VNET, and even from on-premise to VNET via VPN connection. They will not work if you try to cross the Azure Load Balancer (SLB).
Azure Load Balancer (SLB) uses Round-Robin policy to distribute incoming connections?
This is a very common misunderstanding on how SLB works. Actually, Azure SLB is a Layer-4 software Load Balancer and uses a 5-tuple (source IP, source port, destination IP, destination port, protocol type) to calculate the hash that is used to map traffic to the available servers behind a VIP. The hash function is chosen such that the distribution of connections to servers is fairly random then not perfect round-robin. Additionally, at least today, session affinity is not supported.
Microsoft Azure Load Balancing Services
http://azure.microsoft.com/blog/2014/04/08/microsoft-azure-load-balancing-services
In October 2014, Azure introduced a new distribution mode called Source IP Affinity (also known as session affinity or client IP affinity). Azure Load Balancer can be configured to use a 2 tuple (Source IP, Destination IP) or 3 tuple (Source IP, Destination IP, Protocol) to map traffic to the available servers. By using Source IP affinity, connections initiated from the same client computer goes to the same DIP endpoint.
Azure Load Balancer new distribution mode
http://azure.microsoft.com/blog/2014/10/30/azure-load-balancer-new-distribution-mode
Azure Load Balancer (SLB) supports SSL termination?
Azure SLB does not support SSL termination at the edge yet; you have to handle the termination process for HTTPS encryption and decryption within each VM or Web/Worker role instance. This is one of the network improvement areas that is under consideration by the Azure Networking team.
Allow SSL termination at the load balancer
Can I use my own Load Balancer and avoid Azure SLB?
YES, usage of Azure SLB is not mandatory, there is a specific offer from Azure called “Basic” that will give you up to 27% lower prices, compared to Standard SKUs:
Basic Tier Virtual Machines
http://azure.microsoft.com/blog/2014/04/28/basic-tier-virtual-machines-2
Autoscaling will not be available with “Basic”, but you will still have 99,95% through “Availability Set”. Please also notes that not all the sizes, in terms of CPUs and RAM, will be available:
Once selected the “Basic” tier for your VM, you can then install your own Load Balancer software.
Can I expose full port range of my VM?
YES, there is an Azure specific feature called “Instance-Level Public IP Addresses” that will provide your VM a “Public Instance IP” (PIP) with full port range opened by default, without any need to create any endpoint. Using normal VIPs, you can only create/open up to 150 endpoints/ports, then if you need a large number of ports to be opened this is the feature that will satisfy your requirements.
Instance-Level Public IP Addresses
http://azure.microsoft.com/documentation/articles/virtual-networks-instance-level-public-ip
How many connections a VNET joined VM can support?
For Windows Server, a VM can support about 500k TCP connections but you need to be careful about other potential limits that may come into the game before this threshold: if you expose your VM to Internet traffic through an Azure Load Balancer endpoint, you may be limited by SLB capacity or DDOS security mechanisms. Actually, there is no public documentation on SLB or Endpoint limits, neither on the number of connection nor network bandwidth, then you should conduct your own performance tests to ensure everything will work correctly.
Can I have multiple VNET Gateways?
Currently you can only have one network Gateway for single VNET: if you configure multiple tunnels, they will share the same maximum network bandwidth.
As you can see in the table above, there is a limit on the maximum number of tunnels you can create, based on the type of Gateway SKU and connectivity type (S2S VPN or Express Route).
ExpressRoute and VPN can coexist in the same VNET?
YES, Microsoft recently (May 2015) announced the possibility to have a VNET gateway supporting ExpressRoute and VPN at the same time. The only requirement is to enable ExpressRoute before VPN. One Gateway per VNET restriction persists.
New Networking Capabilities for a Consistent, Connected and Hybrid Cloud
Can I connect VNETs across regions, geographies and subscriptions?
YES, Azure provides two ways to connect VNETs: ExpressRoute and VPN. With VPN, there is no boundary on the region, geography or subscription, essentially the tunnel only need Gateway VIPs to see each other. In case you need more bandwidth and lower latency, you
can eventually use ExpressRoute technology to connect up to 100 VNETs (depending on bandwidth) and share circuits between them, even with different subscriptions. Before Premium add-on announced in May 2015, ExpressRoute posed limitations on geographical boundaries: now it is possible not only to connect VNETs in the same geography (US, Europe, etc.) but also across different geographies. For example, you can connect a VNET in West US to a VNET in North Europe.
New Networking Capabilities for a Consistent, Connected and Hybrid Cloud
Does Azure provides DDOS network protection?
YES, Azure infrastructure is designed to protect the network from DDOS originating from the Internet and also internally from other tenants VMs, you can read the details in the white-paper below:
Please be aware of the following important points:
- Windows Azure’s DDoS defense system is designed not only to withstand attacks from the outside, but also from within.
- Windows Azure monitors and detects internally initiated DDoS attacks and removes offending VMs from the network.
- Windows Azure’s DDoS protection also benefits applications. However, it is still possible for applications to be targeted individually. As a result, customers should actively monitor their Windows Azure applications.
Can I use IPv6 inside an Azure Virtual Network?
IPv6 support has been confirmed to be under development as you can read at the link below:
Support IPv6 throughout the Azure Platform
Additional comments from Azure FAQ on http://azure.microsoft.com/en-us/pricing/faq:
Microsoft has played a leading role in helping customers to smoothly transition from IPv4 to IPv6 for the past several years. To date, Microsoft has built IPv6 support into many of its products and solutions like Windows 8 and Windows Server 2012 R2. Microsoft is committed to expanding the worldwide capabilities of the Internet through IPv6 and enabling a variety of valuable and exciting scenarios, including peer-to-peer and mobile applications. The foundational work to enable IPv6 in the Azure environment is well underway. However, we are unable to share a date when IPv6 support will be generally available at this time.
UDP broadcast and multicast are supported inside Azure Virtual Network?
This type of communication is not allowed inside a VNET or even across Azure SLB.
Support Multicast within Virtual Networks
Is it possible to have multiple NICs on VM?
YES, this feature has been announced at TechEd Europe back in October 2014. Be aware that adding more NICs will not give you more bandwidth, all will share the same limit granted to the VM level. Now it's possible to have up to 4 additional NICs without any additional cost, the exact number varies depending on the VM size:
- Large (A3) and A6: 2
- ExtraLarge (A4) and A7: 4
- A9: 2
- D3: 2
- D4: 4
- D13: 4
Be aware that some limitations exist, be sure to read the “Multiple NIC FAQs & Constraints” section in the blog post below:
Multiple VM NICs and Network Virtual Appliances in Azure
http://azure.microsoft.com/blog/2014/10/30/multiple-vm-nics-and-network-virtual-appliances-in-azure
It is worth mentioning that now there is also the possibility to persist the Ethernet (MAC) addresses of the NICs persistent through the life cycle of the VM.
Can I avoid DHCP for my VM inside Virtual Network?
YES, when you join a VM to an Azure Virtual Network, by default it will take an internal IP (DIP) automatically assigned by Azure internal DHCP, valid in the address scope you defined and eventually in the subnet you indicated. This IP will have an infinite lease time, but under
certain conditions may change, for example if you stop your VM with de-provisioning (default behavior). If you are going to mix DHCP dynamically assigned IPs and statically assigned IPs, it’s recommended to use separate subnets to avoid potential address clashes. You can read more details in the blog post below:
Static Internal IP Address for Virtual Machines
http://azure.microsoft.com/blog/2014/04/22/static-internal-ip-address-for-virtual-machines
Is it possible to run network penetration tests against my Azure VMs?
YES, you can do that but it’s highly recommended to follow the specific procedure mentioned below, before running the test, otherwise Azure monitoring and defense systems will trigger in and blacklist your connections, IPs and/or VMs. You need to download a from http://download.microsoft.com/download/C/A/1/CA1E438E-CE2F-4659-B1C9-CB14917136B3/Penetration%20Test%20Questionnaire.docx , fill-in the required information and then open a support ticket to Azure Customer Support and specify "Support Type: Billing", "Problem type: Legal and Compliance" and "Category: Request for penetration testing":
Which is the guaranteed bandwidth for Azure VPN?
There is no minimum guaranteed bandwidth for Azure VPN, the only SLA provided is about high-availability (99,90%) and you can download the related document from the link below. If you search over public content in Internet, you may find several sources reporting various values for max bandwidth, starting from 60 to 100Mbit/sec: this is what I normally obtained with my personal tests, let me emphasize again that there is no minimum guaranteed. The max cap seems to be dictated by the VM size supporting the Azure side of the VPN software (Small size), I heard rumors about future enhancements but nothing official yet.
Microsoft Azure Cloud Services, Virtual Machines, and Virtual Network SLA
http://www.microsoft.com/en-us/download/details.aspx?id=38427
VHD I/O will count toward VM network bandwidth cap?
NO. I/O related to IaaS VM VHDs access will count against storage IOPS limits, not network limits. Seems a tricky question, but since persistent VM storage is networked, some customers and partners frequently ask it. Official numbers on scalability targets for Azure Storage are reported at the link below:
Azure Storage Scalability and Performance Targets
http://msdn.microsoft.com/library/azure/dn249410.aspx
If you use some special features like SQL Server 2014 Azure Blob Storage integration or Azure Files (over SMB), it will count toward your VM network bandwidth limits, in addition to the target service scalability and performance limits.
UPDATED: New White-Paper on SQL Server 2014 and Azure Blob storage integration
Introducing Microsoft Azure File Service
Can I use A8/A9 VM Infiniband NIC for my application traffic?
It depends. I have been asked several time if, for example, Infiniband NIC can be used for SQL Server AlwaysOn Availability Group replication traffic, since it provides very high-bandwidth, low-latency connection and RDMA support. In this specific case, the answer is NO, this NIC does not provide general TCP/IP connectivity, as for all the applications that cannot talk over “Network Direct” interface and MS-MPI protocol.
New High Performance Capabilities for Windows Azure
Can I create a DMZ in Azure?
YES. With the introduction of “Network Security Groups” (NSG) and “Multiple NICs” at TechEd Europe 2014 and “Custom Routes” and “IP Forwarding” at Ignite 2015, it is now possible to implement a full-featured DMZ with strong segregation of traffic and full control between subnets inside a VNET.
There is already a certain number of 3rd-party Network Virtual Appliances (NVAs) you can use to provide security features like firewalls/DMZ, but also IDS/IPS, you can find them in the Azure Market Place through "Azure Certified Program" to ensure quality and simplify deployment.
Azure ILB can be used to augment networking security?
YES, but it’s necessary to clarify what is the security benefit since there is some confusion on how this feature works. Essentially, it can help security since will let you define one (or more) load-balanced endpoints, as when using Azure SLB, not exposed to the Internet, then more secure. Conversely, it’s not an isolation mechanism since will not segregate resources inside a VNET, all the VMs will have open and full connectivity to all other VMs if not using ILB, even if ACLs will be used. For the same reason, ILB cannot be used to realize a DMZ.
If you want to build a real DMZ configuration, it is recommended to look at the previous “Can I create a DMZ in Azure?” section in this blog post.
Can I include Azure PaaS services inside a Virtual Network?
YES. Microsoft recently began to add platform PaaS services addition to VNET: doing this, you will be able to have bi-directional access between your IaaS VMs and PaaS services. Now, you can add Web Sites and HDInsight HBASE clusters to Azure VNET, hopefully will have more in the future:
Azure Websites Virtual Network Integration
http://azure.microsoft.com/blog/2014/09/15/azure-websites-virtual-network-integration
Provision HBase clusters on Azure Virtual Network
http://azure.microsoft.com/en-us/documentation/articles/hdinsight-hbase-provision-vnet
Comments
Anonymous
January 12, 2015
So after a very long read, I'm still none the wiser to know whether Azure provides performance-based SLAs>?Anonymous
January 16, 2015
The comment has been removedAnonymous
March 30, 2015
Hope article is little OLD but really very informative thanks IgorAnonymous
June 10, 2015
NIce job here! Thanks!!Anonymous
June 14, 2015
Great blog, Igor. Thanks.Anonymous
October 15, 2015
> I tested VM bandwidth limits on almost all VM sizes and found the results pretty consistent over time. Since official numbers have not been disclosed yet, I will not share my findings. Terrific 'answer', thanks a lof for it, it was very informative. You're typical Microsoft user indeed.Anonymous
October 17, 2015
The comment has been removedAnonymous
April 22, 2016
To enable RSS by script, you may use this:https://github.com/chgeuer/Quizzer/blob/6fd33acb54d3a51ed9665285d5657c69cf635bab/Quizzer.Web/SetupScripts/install2.ps1#L190-L216Anonymous
January 27, 2017
Hi,currently we have a StorSimple 8000 series. File shares hosted on StorSimple gets replicated to cloud over ExpressRoute link of 2Mbps. But now we We are planning to migrate from on-prem to Azure. Please let me know, if there is any specific ExpressRoute network bandwidth calculator for File Shares?Thanks & Regards..Ujjal- Anonymous
January 27, 2017
Not at my knowledge, I have some expertise on ER but I'm not a super-expert.Regards.
- Anonymous