Features and behavior of Load Tests containing unit tests in VSTS 2008
This article describes the behavior in VS 2008 for load tests that contain unit tests (VS 2005 was slightly different in some respects).
Instances of unit test classes in a load test
Each time an iteration of a unit test is run in a load test, a new instance of the unit test class is instantiated. One of the benefits of this for the unit test author is that even though multiple threads may be running your unit tests in parallel during a load test, you don’t need to synchronize access to member variables of your unit test class (but you do need to worry about this for any static variables in your unit test class).
TestInitialize & ClassInitialize methods
The TestInitialize and TestCleanup methods are run for each iteration of a unit test in a load test.
The ClassInitialize method is run just once (before the first iteration of a test method in the class).
Timers / Transactions
Within a unit test, you can use the BeginTimer() and EndTimer() methods to time portions of the unit test execution, and these results are reported in the Transactions table in the Load Test Results Analyzer.
For an example, see Sean Lumley’s blog on Load Testing Web Services with Unit Tests: https://blogs.msdn.com/slumley/pages/load-testing-web-services-with-unit-tests.aspx
Note: There was a bug in VS 2005 where timer results were not reported for data-bound unit tests included in a load test. This is fixed in VS 2008.
Data binding
Each time an iteration of a data-bound unit test is started in a load test it gets the next row from the data source (either sequentially or randomly depending on the DataAccessMethod specified in the DataSource attribute for the unit test method). The iteration of the unit test runs using that data row and a pass/fail result (and timing) for that unit test iteration is reported in the load test results. If the number of unit test iterations run in the load test exceeds the number of rows in the data source, rows from the data source are used multiple times (again either randomly, or sequentially after cycling back to the start depending on the attribute).
The data binding is independent of and not affected by the threading or user models described below.
Threading
For a load test that contains unit tests, a thread pool is created when the load test starts. The thread pool has a min and mix size determined by the load pattern for the Scenario in the load test. For a constant load pattern, the min and max size of the thread pool are the same as the constant user load. For a step load pattern or goal-based load pattern, the min and max size of the thread pool are different based on the properties for these load patterns. When the load test starts, before running any unit tests, the thread pool is created with the min number of threads ready to run. Unit tests are run on free threads in the thread pool. When a unit test completes the thread becomes available for re-use, so the next unit test to start may run on the same thread. If the user load grows larger than the min thread pool size, new threads are created and added to the thread pool (up to the max thread pool size which was based on the max user load).
The point of this explanation is that virtual users in a load test are not exactly the same as threads. A particular unit test iterations runs to completion on a single thread, but then the thread may be reused for unit tests on behalf of different virtual users.
Passing Load Test Context Parameters to Unit Tests
Sean Lumley has a blog post that demonstrates how to pass load test context parameters to a unit test: https://blogs.msdn.com/slumley/pages/passing-load-test-context-parameters-to-unit-tests.aspx. This may be useful if you have some variables that you want to be able to change for each running of a load test. His simple example shows you how to create a load test plug-in which will read the load test context parameters and pass them on to the unit test.
Per user data
If you need to keep data for each virtual user in a load test containing a unit test, the Load Test framework in VS 2008 now has a LoadTestUserContext class that helps with this. Here’s an example unit test that does this. I recommend that you create your own class to hold the per user data as shown in the example. For one thing, this method allows you to run the unit test in a load test or by itself.
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.VisualStudio.TestTools.LoadTesting;
namespace TestProject1
{
internal class MyUserData
{
private int m_userId;
private int m_userUseCount;
public MyUserData(int userId)
{
m_userId = userId;
}
public int UserId
{
get { return m_userId; }
}
public int UserUseCount
{
set { m_userUseCount = value; }
get { return m_userUseCount; }
}
}
[TestClass]
public class UnitTestClass
{
public UnitTestClass()
{
}
private TestContext testContextInstance;
public TestContext TestContext
{
get
{
return testContextInstance;
}
set
{
testContextInstance = value;
}
}
private MyUserData GetUser()
{
MyUserData user;
if (this.TestContext.Properties.Contains("$LoadTestUserContext"))
{
// This case occurs when the unit test is run inside a load test
// Get the LoadTestUserContext
LoadTestUserContext loadTestUserContext = this.TestContext.Properties["$LoadTestUserContext"] as LoadTestUserContext;
// Get MyUserData object from the LoadTestUserContext object, or create and add it if not already there
if (loadTestUserContext.ContainsKey("MyUser"))
{
user = (MyUserData)loadTestUserContext["MyUser"];
}
else
{
user = new MyUserData(loadTestUserContext.UserId);
loadTestUserContext.Add("MyUser", user);
}
}
else
{
// This case occurs when the unit test is run outside of a load test; just need one user
user = new MyUserData(1);
}
return user;
}
[TestMethod]
public void PerUserDataExample()
{
MyUserData user = GetUser();
user.UserUseCount++;
// Fail once the user object has been used more than 10 times (just to demo re-use of MyUserData)
Assert.IsTrue(user.UserUseCount <= 10, "Use count for MyUserData object: " + user.UserUseCount.ToString());
}
}
}
Try creating a load test in VS 2008 that contains this unit test with the following settings:
User Load: Constant 5
Test Iterations: 100
Percentage of New Users: 0
With these load test settings, each of the 5 virtual users should run 20 test iterations on average, and you should see 50 of the 100 test iterations fail due to the assert that UserUseCount < 10.
If you change the Scenario property "Percentage of New Users" to 100 (which was the default in VS 2005), then a new user context is created for each iteration of the unit test in the load test and user.UseCount will never exceed 1, and the assert will never fail.
The online doc page describing the LoadTestUserContext is at: https://msdn2.microsoft.com/en-us/library/microsoft.visualstudio.testtools.loadtesting.loadtestusercontext(VS.90).aspx. Unfortunately, this page does not yet contain descriptions for the methods and properties, but many of them will be obvious by the names and signatures, and the descriptions will be available with the RTM release of VS 2008.
App Domains
When a unit test is run by itself, a separate application domain is created in the test process for each unit test assembly. There is some overhead associated with marshalling tests and test results across the application domain boundary, so when running unit tests in a load test, the application domain is not created by default. This provides some performance boost in terms of the number of tests per second that the test process can execute before running out of CPU. The only drawback is that if the unit test depends on an app.config file, this doesn’t work without creating the app domain. In this case, you can enable the creation of app domain for the unit tests: in the Load Test editor’s Run Setting’s properties set the property “Run unit tests in application domain” to True.
Performance Improvements from VS 2005 to VS 2008
The performance of running unit tests in a load test has been improved in VS 2008, and is especially noticeable for unit tests with a very short duration.
Comments
- Anonymous
October 12, 2007
I've posted an article that describes how Load Tests that contains Unit Tests work in VS 2008: http://blogs.msdn.com/billbar/pages/features-and-behavior-of-load-tests-containing-unit-tests-in-vsts-2008.aspx - Anonymous
October 12, 2007
I've posted an article that describes how Load Tests that contains Unit Tests work in VS 2008: http - Anonymous
March 17, 2008
Visual Studio Team System for Testers Content Index for Web Tests and Load Tests Getting Started Online - Anonymous
May 19, 2008
Load-Test mit VSTS und app.config - Anonymous
April 21, 2010
There are some very great sources here and thank you for being so kind to post them here.