On Clouds, same CPUs deliver the same Performance

A Blog Post on Internet pretends that Azure VMs are 3 to 4 times slower than Amazon's

Two weeks ago, one of my customers told me that he saw a blog post on the net saying that Azure virtual machines are 3 to 4 times slower than their equivalent on Amazon Web Services. The blog post pretends that the tests were impartial and targeted on CPU performances.

Strange isn't it? Normally, the same CPU delivers the same performance if the test is really objective. There may be a small difference, due to the hypervisor overhead; but 4 times slower is really huge.

So I decided to do some investigations to validate the numbers.

Here is the blog post : https://www.pythian.com/blog/comparing-cpu-throughput-of-azure-and-aws-ec2/

The guy, Marc Fielding, said that he compared an Amazon M3.XLarge with an Azure A3 Standard. Both VMs were running Oracle Linux. The blob post has a couple of comments and those who commented asked Marc to run sysbench on the VMs to compare. Results are :

AWS M3.XLarge (E5-2670 @ 2.5GHz) : Sysbench results = 51.64sec

Azure A3 (E5-2660 @ 2.2GHz) : Sysbench results = 117.08sec

Wouaou ! That a huge difference. There should be a difference has this is not the same processor nor the same speed, but still, it's a lot.

My own tests

Let me do my own tests. I used Oracle Linux 7, instantiated on an Azure A3 VM on West Europe DataCenter:

 

Sysbench is not installed by default, so I did install sysbench in three steps :

 

yum install wget

wget https://dl.fedoraproject.org/pub/epel/7/x86\_64/s/sysbench-0.4.12-12.el7.x86\_64.rpm

yum install sysbench-0.4.12-10.el7.nux.x86_64.rpm

Then let's start the tests.

Tests on an Azure A3 virtual Machine (on Amsterdam DC)

First, is an A3 really hosted on an E5-2660 @ 2.2GHz ??

Just run cat /proc/cpuinfo

Surprise, an A3 is not hosted on an E5-2660, but on an AMD Opteron 4171 HE @ 2.1GHz So we should expect a less good score than the AWS M3.XLarge. Let 's see:

sysbench –-test=cpu –-cpu-max-prime=50000 –-num-threads=2 run

Indeed, we do have a less good score, but that is not 117sec as described in the blog post.

Let's do more tests on different Azure Virtual Machines Sizes

Resizing the VM to D3 (on Amsterdam DC)

So what is the CPU on a D3 ?

Good!! The CPU is not exactly the same than the AWS one, but the D series is hosted on an Intel proc.

Better now, we get 71.59sec on a bit older proc and lower frequency. But that would be great to have EXACTLY the same CPU.

Resizing the VM to A11 (on Amsterdam DC)

A10 and A11 and "compute intensive" virtual machines, without infiniband support. They should be fast.

cat /proc/cpuinfo

YYEESSS! Same CPU, same frequency. Normally, this VM should deliver exactly the same CPU performance than the AWS one. Suspense…

Yes! 51.21 sec. Demonstration is done. Same CPU, same results.

Just for fun, let's give try on our Godzilla Virtual Machines. The G-Series…

Test on a G2 Series (West US DataCenter)

The G-Series has the fastest CPUs on Azure:

The frequency is not high, but the architecture is a bit more performant. Here are the sysbench results:

As expected, faster proc = better results.

Conclusion

I honestly don't know how the performance tests have been done to produce the blog post mentioned at the beginning of this article. Different CPUs produces different performance tests results, and identical CPUs produce the same performance results. There is an overhead due to the Hypervisor, but tier 1 Hypervisors are equivalent and introduce a negligible overhead in such testing. In term is pure CPU performance, the same CPU at the same frequency will deliver the same performance whether it is an Azure VM, or and AWS or any other cloud provider, considering the fact that the hypervisor is performant.

In summary

Here are the results of my tests:

Oracle Linux 7.0 with Sysbench 4.12

To display the CPU architecture: cat /proc/cpuinfo

To launch the bench: sysbench –-test=cpu –-cpu-max-prime=50000 –-num-threads=2 run

Azure VM Size	CPU @ Frequency	Sysbench Result
A3	AMD Opteron 4171 HE @ 2.1GHz	88.44 sec
D3	Intel Xeon E5-2660 @ 2.2GHz	71.59 sec
A11	Intel Xeon E5-2670 @ 2.6GHz	51.21 sec
G2	Intel Xeon E5-2698B v3 @ 2.0GHz	46.03 sec