Test agent, Test Controller and MSTEST FAQ
MSTEST:
1. Do I need to install full VS to use MSTEST? Is there any light weight option?
You need not have full VS to bring in MSTEST. Test agent is a light weight option to bring in MSTEST. Refer to the below link for details:
2. Does MSTEST work for 64 bit binaries? Any known issues with it?
MSTest doesn’t work for 64bit binaries. However mstest can be used for 64bit testing. To do this, test assembly needs to be compiled for “AnyCPU” and test setting should specify 64bit host process for test execution. Also the test should be running on 64bit machine. This will load the test assembly in 64bit and testing will happen in 64bit mode.
3. Does Side by side installations work?
For VS:
VS 2008 and VS 2010 on same machine, works fine.
For test agent and test controllers:
For agent and controllers, side by side install is not supported.
4. What kind of logs do I need to debug issue with a test run?
For VS 2010, the devenv logs and QTAgent.exe logs are needed.
In VS 2008, you need to enable the logs for VSTESTHOST process:
https://blogs.msdn.com/yutong/archive/2006/10/03/How-to-Enable-VSTS-Logging_3F00_.aspx
Also take out the logs from the event viewer under event source “VSTTAGENTProcess”
5. Publish issues with MSTEST
VS 2008 and TFS 2008:
Error:
Publishing results of test run geobuild@B88PSRACK017 2010-05-05 10:47:20_Any CPU_Release to https://tkbgitvstfat04:8080/............Publish completed successfully.
Done executing task "TestToolsTask".
Resolution:
In 2008, MSTest pushes results to the server, which then copies them out to the drop share.
Hence the TFS AT service account (not the TFS *Build* service account) needs permissions to the share.
VS 2010 and TFS 2008
Error:
Unit tests are executed properly, but code coverage is not reported back to the TFS 2008 data warehouse as the build summary says "No Code Coverage Results". I get an additional log file on the build server in ...\TestResults\CoverageLog5.log saying::
The Code Coverage Analysis Service could not analyze the code coverage data for build DevLineDailyBuild>20100630.3 (Any CPU/DevDeploy). This is frequently caused by publishing different versions of the same binary to a particular build.
Processing of code coverage data from run XXX@ServerName 2010-06-30 02:34:05_Any CPU_DevDeploy caused the following exception to be thrown:
Error when creating coverage info: Error loading symbol file. Symbol and binary files should be in the same folder as the coverage file or on the symbol path: C:\Documents and Settings\usdg-andromeda\Local Settings\Temp\Bode\DevLineDailyBuild\Binaries\DevDeploy\XYZ.dll.
The data.coverage file is generated at “TestResults\In\ServerName\” folder. “TestResults\Out” folder is empty.
Resolution:
This a known issue where publishing code coverage from 2010 client to a 2008 server was broken.
6. Is it possible to pass custom parameters to through MSTEST that will be passed to Setup and Cleanup scripts?
Or
My “buildrop” location keeps changing everyday. I need to deploy some binaries from my build drop for my test run. Does the VS unit testing framework supports any token for %builddrop% ?
It is not possible to pass custom parameters through MSTEST. Alternative could be to set the required parameters as environment variables and let you script take those values.
Test execution and VS:
7. What is the order in which initialize and cleanup methods get called?
All cleanups (assembly clean and class cleanup) will be called only during the end of the test run after all the tests in all the classes are executed.
Let’s consider an example: Two test projects: A.dll (class 1 - test 1a, test 1 b, class 2- test 2) and B.dll (class 3- test 3)
Below will be the rough flow of execution:
· Assembly init for A
· Class init for class1
· Test init for Test 1A
· Test 1A
· Test cleanup for test 1A
· Test init for Test 1b
· Test 1b
· Test cleanup for test 1b
· Class init for class 2
· Test init for Test 2
· Test 2
· Test cleanup for test 2
· Assembly init for B
· Class init for class 3
· Test init for Test 3
· Test 3
· Test cleanup for test 3
· Class cleanup for class 1
· Class cleanup for class 2
· Assembly cleanup for A
· Class cleanup for class 3
· Assembly cleanup for B
Below posts explains them in detail:
https://blogs.msdn.com/nnaderi/archive/2007/02/17/explaining-execution-order.aspx
https://blogs.msdn.com/b/ploeh/archive/2007/01/06/classcleanupmayrunlaterthanyouthink.aspx
8. How will I make sure that all the test cases in one class get executed before the tests in another class?
There is no way to guarantee that all tests in one class get called before the tests in another class. One work around to achieve this is to put the tests in the class inside an ordered test. But even with this approach, the class cleanup of this class will not be called right after the execution of the ordered test. If you have to dispose objects before starting execution of tests in another class, you need to add the cleanup logic in test cleanup of the last test method in the ordered class.
9. How can I create my custom test type? Do we have samples for Custom host adapter sample?
VS SDK 2008 downloads: https://www.microsoft.com/downloads/details.aspx?familyid=30402623-93ca-479a-867c-04dc45164f5b&displaylang=en
VS SDK 2010 Beta 2 downloads: https://www.microsoft.com/downloads/details.aspx?FamilyID=cb82d35c-1632-4370-acfb-83c01c2ece24&displaylang=en
10. I get the below error while trying to run my custom test type. “File extension specified '.aii' is not a valid test extension.”
a. Remote execution :
1) Make sure that the host adapter is chosen properly in test settings ( to chose the host adapter in test settings it need to be registered on the machine where you create test settings. If you are running test from Microsoft Test Manager this has to be present in MTM machine as well)
2) Register the test type and host adapters in the test controller machine and the agent machines. And you restart the services after doing the registration.
b. Local execution:
Make sure that the test type is registered properly. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\10.0\EnterpriseTools\QualityTools\TestTypes for VS .
11. "One or more of the tests (directly or indirectly) included in this run could not be found" –
This error happens when no test case can be found with the given ID. In the case ordered test, the contained test method name would have changed. For custom test types, make sure that you are not generated a random GUI for test ID during each build. Instead you can return the hash of Test method name, class name and namespace name as the test ID (similar to how out of box test types work).
12. Any way to out a hyper link or custom sections in the results file?
No.
13. Is there a way to run unit tests in parallel?
Check out https://blogs.msdn.com/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx for enabling parallel execution of unit test. Note that it is up to you to make sure that each individual test in the run can be run side by side with some other test without any conflicts (like cleanup issues, one negative test running in parallel with a positive test…).
Deployment
14. How does deployment work?
Test execution deploys the below items :
· DeploymentItem()] items placed on [TestMethod] function
· Those in the /runconfig file in the Deployment files section
· Those that have reference (strong) from within the binary. This usually requires a reference in code to make sure the meta data is in the assembly
“CopyLocal=true” has no effect on whether MSTest commandline deploys the item
Mstest goes only one level deep to deploy the reference assemblies . If A refers B and B refers C, it wont deploy C.
15. Deployment is very slow
If the _NT_SYMBOL_PATH environment variable points to a non existent path, then deployment might be slow.
Details in this blog post.
16. How does deployment of .config files work?
By default only the config file of the test binary will be deployed and not other config files . More details here. For other .config file, you need to add them as deployment items explicitly.
18. What is the best way to deal with deploying dependent assemblies while running tests?
It is a normal for the test code to take dependency on product binaries. In such cases, the whole product binaries need not be copied to the location of the test binaries.
You can simply set the "Root folder" in the test settings to the location of your product binaries. This path will then act as probing path during assembly resolution.Refer to the this blog post for details on the same :
https://blogs.msdn.com/vstsqualitytools/archive/2010/01/09/assembly-resolution-for-unit- tests.aspx
Note that, whatever binaries that are present in the probing path will be resolved properly, irrespective of whether they are deployed along with the test binaries or not.
19. How to customize the default deployment directory?
You can change the default deployment folder by editing the test settings file in the XML editor:
<Deployment userDeploymentRoot="C:\TestResults" useDefaultDeploymentRoot ="false" />
Note that ,if the test settings is modified by XML edit (instead of using the default editor) VS need to be closed and reopened ( since the editing is done in XML , the changes will not be updated in the loaded settings.)
Test Rig:
20. Is there a way to programmatically control the rig operations?
The controller SDK is not a public one. However you can achieve various rig options programmatically as below:
a) Register agent with controller - TestAgentConfig.exe /Configure ( this command line will be present in %ProgramFiles%Common7/IDE in the agent machine )
b) Start/Stop service - sc \\machinename stop vsttagent ; sc \\machinename start vsttcontroller. If the agent and controller are in different domain, you can use IP instead of machine name.
c) Remove an agent from the controller - TestAgentConfig.exe /delete
Alternatively, you can edit the QTController.config.xml file (present in %programFiles%Visual Studio 2010/Common7/IDE folder in Controller machine) as well to add agents ( and add properties to agent ). The edit of XML can be done programmatically as well. Make sure to restart the controller service after editing the XML.
21. Will Test Agents allow you to impersonate another user? Or does the Test Agent always kick of tests as the user its configured start up as?
There is no built in support for impersonation, so by default tests run under the identity of the test agent. If this is something you really need, you could write a unit test extension that does the impersonation for you.
22. Agent doesn’t go to ready state or any issue with controller-agent configuration
Refer to Test agent-controller trouble shooting guide
The test controller, test agent logs and the event viewer error messages in the agent/controller machine will also give an idea.
23. How can I distribute my tests across agents?
You can achieve this by setting the bucket size in the test settings file. Refer to this blog for details on it:
https://blogs.msdn.com/b/vstsloadtestblog/archive/2006/02/08/527597.aspx
If you are running tests from VS, then the test settings file can be edited using an XML editor. But for the test settings created using Microsoft Test Manager, there is no easy way to modify the bucket size. This is something we are planning to fix in next release.
24. What happens if I queue multiple test runs on a test rig at the same time?
It depends on the number of available agents in the controller. Consider there are three agents which matches the criteria. The first run that gets queued will reserve ExecutionAgent1. And the second run shall use EA2 and the third will pick up EA3. The fourth run will fail with error mentioning “No agent found …”. But if the count of test cases in the first run is > the bucket size ( which is 100 by default), then appropriate number of execution agents are reserved. For example: If the first run has around 250 test cases, then all three agents will be reserved by the first run. 100 tests will send to ExecutionAgent1 and ExecutionAgent2 and 50 tests will be sent to ExecutionAgent3. And the second run will fail with “no agent …” error.
25. I have successfully deployed test agents and a test controller. The agents show up healthy when I ‘manage test controllers’ from VS. However when I try and use the agents to run a test case VS just shows the test results as Pending and it never completes.
It might be because of blocking firewall in the VS client machine.
VS client doesn’t listen on one specific port for the incoming calls from controller. The port will be chosen in random (default range 1000-5000 or so). Hence you need to add the QTAgent.exe process (instead of port) in firewall exception list in the client machine.