How stable is the performance of Azure VMs?

Question

I need to do automated performance tests of new releases of a web application. The purpose of the tests is primarily to catch performance regressions before a new release goes live.

The production environment isn't in the cloud but we have test environments running on Azure VMs.

We're considering running our tests on Azure VMs but we're concerned that the VM performance is too variable to allow us to measure changes in the application.

Does anyone know how stable Azure VM (mostly cpu but also disks, etc.) performance is hour-to-hour/day-to-day/month-to-month? Am I likely to have difficulties comparing the performance of multiple test runs on the same VMs?

Answer 1

Good day Simon @SimonLarsen-0950

I think that this is an awesome question! I see many people which make the mistake of using shared environment for performance test without understanding the potential issues.

This is my personal 2 cents:

To the lazy readers: The short answer is that as long you use shared resources with other client, there cannot be a guarantee that you will have the resources at all time. Therefore, comparing behavior between different ranges of time cannot provide the exact precision.

The good news is that Microsoft Provides a service of dedicated server, or as the name of the service: Azure Dedicated Host.

The bad news is that the **prices** are extremely high (in my opinion), and personally I did not find any scenario that this service will fit for testing yet.

So, what next? What can we do if we have to build a testing environment for comparing performance?

In this case the answer to the question "Does anyone know how stable Azure VM (mostly cpu but also disks, etc.) ", is NO! There is no choice but to compromise on the precision of the test.

Not even Microsoft can guarantee since no one can know what and what the other users on the same host will use more resources.

The only thing we can do is to use common sense and try to get the best stable tests.

1) Always make sure that you use the exact same size, and pricing model (series), and location for your VM

2) Prefer to test when you think that others do not use a lot of performance

2.1) recommended to do the tests in different hours and days so you will have more understanding on the general behavior of the resources (does not guarantee since we cannot be sure if our VM moved to another host or if the other users on the same host where not changed but this is what we can do

Not... The forum prevent me from posting a longer answer, so I have to stop here. I think I covered the main important points