Running Web and Load tests from the Command-Line
This blog post will show you how to run a web test or load test from the command-line instead of from within Visual Studio. Most of the time you will want to run you web or load test from within visual studio, but you can run them from a command-line. Maybe you have a set of 5 web tests that you want to run once a day. You could configure a scheduled task to run a batch file which contained ran the 5 web tests. The command line executable that is used to run tests is called mstest.exe and is installed when you install either VS Team Suite or VS Team Test SKU. Here is a MSDN help link for mstest: https://msdn.microsoft.com/en-us/library/ms182486.aspx
The easiest way to get started with mstest is to start a visual studio command prompt. Go to Start Menu -> All programs -> Microsoft Visual Studio v9.0 -> Visual Studio Tools -> Visual Studio 9.0 Command Prompt. This will launch a command window which has mstest.exe in its path. So type mstest /? This will show each of the available options. Let’s run through a few options of how to run tests.
1) Running a single WebTest
First let’s run a simple web test. To do this, we just need to specify the /TestContainer argument. A .webtest file or .loadtest file is considered a test container in the same way that a dll which contains unit tests is a test container. So if we have a web test called WebTest1.webtest, you would use the following command to run the web test:
mstest /TestContainer:WebTest1.webtest
This will produce output similar to the following:
Loading WebTest1.webtest...
Starting execution...
Results Top Level Tests
------- ---------------
Passed webtest1.webtest
1/1 test(s) Passed
Summary
-------
Test Run Completed.
Passed 1
---------
Total 1
Results file: C:\Documents and Settings\slumley\My Documents\Visual Studio 10\Projects\TestProject3\TestProject3\TestResults\slumley_SLUMLEYTCM 2008-12-22 13_38_14.trx
2) Running Multiple WebTests
Now let’s execute 2 web tests from one call to mstest. You can specify multiple test container arguments on one command-line. So say we wanted to execute WebTest1.webtest and WebTest2.webtest. You would use the following command-line:
mstest /TestContainer:WebTest1.webtest /TestContainer:WebTest2.webtest
This will produce output similar to the following:
Loading WebTest1.webtest...
Loading WebTest2.webtest...
Starting execution...
Results Top Level Tests
------- ---------------
Passed webtest1.webtest
Passed webtest2.webtest
2/2 test(s) Passed
Summary
-------
Test Run Completed.
Passed 2
---------
Total 2
Results file: C:\Documents and Settings\slumley\My Documents\Visual Studio 10\P
rojects\TestProject3\TestProject3\TestResults\slumley_SLUMLEYTCM 2008-12-22 13_4
3_04.trx
3) Specifying deployment items
One of the big differences with running from the command line is that you are not within a visual studio project. When you run a web test from within Visual Studio, we will try to determine what needs to be deployed with the web test in order for it to run. For example, maybe you have written a custom validation rule or a extraction rule. When you run the test from within visual studio, we will look at the references of the test project and try to figure out what needs to be deployed.
When you are running from the command-line, you will need to be very explicit about what you deploy. So if you have a dll that needs to be deployed for a test to run, you will need to inform mstest. The way you do this is by specifying /runconfig parameter. One of the sections of a run config is deployment items. Use visual studio to create a run config. Then to run a test with a runconfig, you would have a command-line similar to the following:
Mstest /TestContainer:WebTest1.webtest /RunConfig:NewRunConfig.testrunconfig
4) Executing a test on a controller/agent setup
When in VS, to run a web or load test on a controller/agent setup you need to use a test run configuration. You would open the run config, specify to run the test remotely and specify a controller name. It works the same way from the command-line. You will need to create a run config that has the correct settings. Then you run it the same way as you do when specifying deployment items:
Mstest /TestContainer:WebTest1.webtest /RunConfig:NewRunConfig.testrunconfig
5) Running a coded web test
Running a coded web test is similar to running a unit test. You will set the /testcontainer argument to the dll which contains the coded test.
mstest /TestContainer:TestProject1.dll
One thing you will notice when running a test this way, is that mstest will execute all of the tests within the dll.
6) Executing a specific coded test in a dll
So if you want to execute just one test within a dll, you would need to use the /test argument. For example, to run WebTest1Coded, you would use the following command line:
mstest /TestContainer:TestProject1.dll /Test:WebTest1Coded
If you wanted to execute 2 tests, you can specify multiple /Test arguments. For example:
mstest /TestContainer:TestProject1.dll /Test:WebTest1Coded /Test:WebTest2Coded
7) Executing a load test
This is the same as executing a web test except you need to specify the .loadtest file
mstest /TestContainer:LoadTest1.loadtest
When the test completes, you can either open the trx file to get to the load test results or you can use the manage load test results dialog form within VS. Here is a blog on that: https://blogs.msdn.com/slumley/pages/managing-load-test-results.aspx
8) Specifying the results file name
You will notice that the results file(trx file) gets a unique name which contains user, machine and a timestamp. If you want to specify the name of the results file and where it is generated, you can use the /resultsfile parameter. For example, to create a results file called MyResults.trx and have it saved to c:\results, I would do the following:
mstest /TestContainer:WebTest1.webtest /resultsfile:c:\results\MyResults.trx
Hopefully this post shows you how to make use of the mstest tool.
Comments
Anonymous
December 22, 2008
The comment has been removedAnonymous
December 23, 2008
This blog just summarises some of the issues I picked up … some of which ‘again’ … during discussionsAnonymous
December 25, 2008
This is a very helpful blog. How about the /searchpathroot switch? Team foundation build passes this switch to MSBuild.exe, while this switch is not documented.