Share via

Environment Agnostic Tests

  • Article

What is an environment agnostic test?
This is a test which will execute successfully no matter how the environment is configured assuming that the product which you intend to test is available on the machine.

What are some possible environment variables?
Operating system, platform, OS language, product language, product sku, product settings, additional applications, etc.

As seen here there are many variables and one thing that my team expects is that our tests will execute the exact same way under all circumstances providing that the test is executing in a supported scenario. This leads to a rather large automation matrix that must be tested against before we ship (this will be another post), but it also leads to a very well defined and efficient test infrastructure.

The above expectation sparked a discussion because we were executing a joint run that included both VB IDE and C# IDE tests, which resulted in some of the C# tests failing. They were failing because in order to execute a joint run with both VB and C# tests, all tests must execute under the VB profile. All tests must execute under this profile, because the VB tests are not environment agnostic (yet!) and all of the C# tests are (well should be).

We will be discussing this topic more in our team meeting on Monday in order to assure we are all on the same page and I will make it one of my missions over the next few months to ensure my teams tests will execute flawlessly under any supported circumstances.

My questions for you are what kind of tests do you execute? What expectations do you have for your environment? Have you had similar discussions at your company?

Comments