Distributed Functional Testing with Visual Studio 2005 Team Test Load Agent
Distributed Functional Testing with Visual Studio 2005 Team System
Applies to:
Visual Studio 2005 Team Edition for Software Testers
Visual Studio 2005 Team Test Load Agent
Summary: This article describes the features of Microsoft Visual Studio Team Test Load Agents that can be used for distributed functional testing.
Introduction
Belying its name, Team Test Load Agents were designed to run both functional tests and distributed load tests. It is easy to set up a “rig” consisting of a controller and set of test agents. The agents can then be used both for distributed load testing and for distributed functional testing.
The key features that agents provide for load testing are:
• Scale out load generation
• Efficient load generation (1000 users from a typical machine running a typical web test)
• IP Switching for testing web farms
• Per CPU licensing (no vUser licensing)
• Performance counters collected on Load Agents and machines under test
• Data (stats and logs) rolled up in central SQL Server for analysis
• Load Test Monitor enables monitoring a test in progress
The key features agents provide for running unit and functional tests are:
• Parallel distributed functional test execution
– Run a large set of unit and functional tests at the same time
– Agent selection enables matrix testing
– Code Coverage
– Better quality of service: test restart if test process dies
• Run tests from VS or Command Line
• Deployment of test code, reverse deployment of logs/test results
• Share test lab hardware with a team
– Multiple clients can submit test runs
– Controller queues runs and manages agents
• Platform for test tools
– Extensible architecture supports third party test types
• Team Foundation Server integration
– Reporting
– Work Item tracking
– Build Server
– Source Code Control
Visual Studio Team Test Edition can execute tests locally, which can cover many needs of verifying product code.
This article focuses on the feature that can be used to perform distributed functional and unit testing on a rig.
Test Run Configuration
Test Run Configurations are files that keep different settings for how you wish to execute tests. There are many settings that can affect the data collected and how the test will be executed. The focus here will be on settings that can affect the execution of a test run: timeout values, executing locally, executing remotely, setup and cleanup scripts, executing with specific agent machines, and deployment.
You can have many configuration files within a single solution. This allows you to quickly select a group of settings for a set of given tests. Any Test Run Configuration can be used for any selected set of tests.
The Test Run Configuration dialog is accessed from the ‘Test’ menu. Selecting the ‘Test’ menu and clicking on the ‘Edit Test Runs Configurations’ one or more available Test Run configuration files will be available.
Local and Remote
Selecting ‘Local’ the test will run in a separate test process called VSTestHost.exe on the same machine as VS. VSTestHost.exe will run under the same user account running Visual Studio. Running locally allows you to do much the same testing you would do in a remote environment.
Some things to note:
· With this configuration the execution of tests is affected by any locally running processes, which may skew results.
· You can not disconnect from a test and reconnect at a later time. That is, if you stop VS the test will stop.
· You can do User Interface (UI) testing.
· If the test crashes VSTestHost.exe there is no restart functionality.
Selecting ‘Remote’ you need to also type in the IP or computer name of the machine the Microsoft Visual Studio Team Load Agent controller has been installed. This controller may have one or more agents it manages. You will see later how to determine what agent it manages. The controller may reside on the local machine or may be in a remote lab (ensure your firewall has exceptions for Visual Studio and the controller so you can connect to remote machines). You can also specify agent properties to select which agents to run your tests on. We will cover this feature in more depth later.
Once you select a controller and save the Test Run Configuration any tests executed using this Test Run Configuration will be sent to the specified controller using the specified settings.
Deployment/Reverse Deployment
Deployment allows you to select files you want copied to the working directory on the agent(s) and available for your tests to use during execution. Bring up the Test Run Configuration dialog. Click on ‘Deployment’ in the left pane. You can add a single file by clicking ‘Add File…’ or add all the files in a given directory by clicking ‘Add Directory…’ Target assemblies are deployed using this mechanism. This is useful if you have a special file containing data you need for the product code (also referred to as code under test) or data files for your tests (say a file with a list of users).
You can find the file location using the test context passed into your test.
string deployDir = this .TestContext.TestDeploymentDir;
This will return a path to the deployment files. Add your file name to the path returned and you can open your file for reading.
Deploying files from the agent(s) back to the local machine is called Reverse Deployment. To have a generated file returned from the agent(s) create or copy your file at the end of the test execution to the path returned by this variable:
string logDir = this .TestContext.TestLogsDir;
Setup and Cleanup Scripts
Setup and Cleanup scripts are deployed to agent(s) and executed before execution of a test run and/or at the end of a test run respectively. Bring up the Test Run Configuration dialog. Click on ‘Setup and Cleanup Scripts’ in the left pane. To specify a setup or cleanup script type in the path to a *.bat or *.cmd file or click on the appropriate ‘…’ and select the file in the file list.
The scripts are sent as strings to the agent(s). The agent(s) take the string and generate a setup (qtsetup.bat) and cleanup (qtcleanup.bat) script file as appropriate, adding test run specific environment variables before executing your commands. For example, say you specify myCleanup.cmd as the cleanup script.
myCleanup.cmd:
echo CleanupScript
echo %AgentName%
The final script on the agent will look like this:
REM ****************************************************************************
REM * This is a generated file by Quality Tools Agent Service
REM * Copyright(c) Microsoft Corporation 2004
REM *
REM ****************************************************************************
set TotalAgents=1
set ControllerName=ERICNWA2
set AgentName=ERICNWA2
set AgentWeighting=100
set TestLogsDir=D:\Documents and Settings\LocalService.NT AUTHORITY.000\Local Settings\Application Data\VSEQT\QTAgent\d93be121-5407-47ce-84b1-61b52dbec744\Results
set TestDeploymentDir=D:\Documents and Settings\LocalService.NT AUTHORITY.000\Local Settings\Application Data\VSEQT\QTAgent\d93be121-5407-47ce-84b1-61b52dbec744\Deployment
set AgentId=1
set TestDir=D:\Documents and Settings\LocalService.NT AUTHORITY.000\Local Settings\Application Data\VSEQT\QTAgent\d93be121-5407-47ce-84b1-61b52dbec744
REM ****************************************************************************
REM * User Commands
REM *
REM ****************************************************************************
echo CleanupScript
echo %AgentName%
Notice the executed cleanup script in this example contained the two ‘echo’ lines which are the ‘User Commands’ sections above. These are the script commands from myCleanup.cmd. The environment variables in the first section of the script file (shown in blue) are test run specific. Meaning they are only valid for that test run, other test runs will have a different generated ID. A setup script would have the same environment variables set at the top of the script as the example cleanup script.
Note that these variables are not available when the tests is actually executing, they are only available during the execution of the user script.
Agent Selection
Agent Selection is one of the more powerful features of the controller. By entering constraints on what properties an agent must have for the given Test Run Configuration you get a lot of flexibility utilizing resources. Some example selection criteria:
· Only use agents that have Windows XP sp2
· Only select agents that have software package Y installed
· Only use agents that belong to my group
· Use all Windows 2000 machines no matter what SP is installed
It was mentioned above when you selected a controller in the Test Run Configuration, you could also specify agent properties. Click on the ‘Test’ menu and click ‘Edit Test Run Configurations’ and select a run configuration. The test run configuration dialog will come up. Select ‘Controller and Agent’ in the left pane.
The agent properties grid is unavailable if a remote controller is not selected. Select remote if it is not.
The grid represents name value pairs each agent must have defined to be selected for executing a test run using this test run configuration.
Say we have the grid defined as follows:
OS WindowsXp
SP SP1
Group A
When a test run is submitted using this test run configuration, any agent with the same above properties assigned will be selected to execute the test run. Not necessarily all of them are used. All the properties specified above must exactly match those defined for each agent; otherwise, the agent is not selected. If the agent only has two of the above name value pairs defined, it will not be a selectable agent.
You can have policy that test runs submitted by a certain group must have A, B, or C defined for the ‘Group’. This would allow multiple teams to share a agent lab through a single controller. The above example is stating that the agent must have a property name of ‘OS’ and it’s value must be ‘WindowsXP’, AND it must have a property defined ‘SP’ with a value of ‘SP1’, finally the agent must have a property, ‘Group’ with a value of ‘A’.
You could set up a test run configuration to select all agents having the property named ‘OS’ with a value ‘WindowsXp’. In the ‘Name’ column type ‘OS’ and in the value column, type ‘WindowsXP’.
This is the only row of data needed to get all agent machines with the same property and value. Agents may have many name/value pairs defined, name/values pairs that apply to other test run configurations. In order for them to be selected for a test run they need to match the name/value pairs entered in this dialog grid. You enter the agent name/value pairs in the ‘Administer Test Controller’ dialog. See below for details.
Test Timeouts
Bring up the Test Run Configuration dialog. Click on ‘Timeouts’ in the left pane.
There are two types of time outs available for any given test run; a) total execution time of all tests in a given test run; b) execution time of individual tests. Total execution time of the entire run allows a test run to be aborted if the run exceeds the amount of time specified. Your organization may have a policy that states verification tests must complete in 20 minutes. Any tests that were not executed before this timeout will be marked as ‘not Executed’.
Individual test timeouts will cause an individual test to be aborted if it exceeds the execution time specified. The next test will be started and the run will continue to execute. The default is 5 minutes.
Note: Load tests do not use these timeout values, except in the case a unit test is added to a load test. Unit tests will have the individual timeouts applied if a value is provided in the dialog box.
Administering the Test Controller
Administering the controller and setting agent properties is handled through the ‘Administer Test Controller’ dialog. To access, click on the ‘Test’ menu and select, ‘‘Administer Test Controller’.
Initially this dialog will come up with a controller: <local>. ‘<local>’ is the local machine, not a controller installed on the local machine. You can only delete temporary files from the local machine using this dialog.
Enter the name of a computer that has the Microsoft Visual Studio Team Load Agent 2005 controller installed. If there are not agents assigned to the controller you will get a pop indicating no agents are configured.
Agent Status
The initial dialog shows the agents the controller manages and their status. Possible status values:
Ready: The agent is ready to execute a test
Running test: The agent is currently running a test.
Disconnected: the agent is marked online, but is disconnected. Here might be why:
· The agent service or agent service UI version may not be running
· The agent can not connect to the controller (firewall, security settings)
· The agent may not have restarted yet. Click the ‘Refresh’ button
Offline: the agent is marked offline
Editing Agent Properties
Once you have added an agent to controller, you can set the agent’s properties in the Administer Controller dialog. Agents are automatically added to a controller when the agent is setup. Agent properties are:
Weighting: This affects the load generated on the agent. Each agent is assigned a weighting, faster machines should be given a bigger weighting. A bigger weighting indicates the agent is capable of simulating a larger portion of users for a load test. For instance say you had the following two agents with the specified weightings and the number of users you want to use for my test:
Agent1 Weighting: 50
Agent2 Weighting: 100
Total weighting: 150
Agent1 will get 1/3 of the load, Agent2 will get 2/3.
So if the load = 1000:
Number of users Agent1 would simulate: ((50 * 1000) / 150) = 333
Number of users Agent2 would simulate: ((100 * 1000)/ 150) +1= 667
If all the agents have the same weighting, then the user distribution is the same on each agent.
IP Switching: Will allow agents to cycle through an IPv4 address range during a load test. This allows you to effectively test a web farm that may have a load balancer distributing requests to the backend web servers. If all the requests were coming from a single all IP they would be routed to the same web server each time.
NIC: This is the network interface card to use to add IPv4 addresses. Note: if the machine status is not ‘Ready’ the available NIC cards will not appear. Close this dialog and mark the agent as online and make sure the state is ‘Ready’ and reopen this dialog.
Base Address: Enter the first three segments of you Ipv4 address. For example: 192.168.0
Start Range: This is the starting number to use in the IPv4 address range. Say the value it 10. When the agent begins generating IPv4 addresses with will start with base address and this number: 192.168.0.<start range> = 192.168.0.10
End Range: This is the last number used in the IPv4 address range. Say it is 15. The agent will generate addresses from the Start Range to the End Range:
192.168.0.10
192.168.0.11
…
192.168.0.15
Subnet Mask: The subnet mask to use with the generated IPv4 addresses. For example: 255.255.255.0
Notes: if the static or DHCP address of the machine is in the range specified here, it will cause an error and not all of the addresses will be added to the NIC card. Also note if the IPv4 address specified is not routable in your network, you will not see any load generated by the client.
Attributes: These are name/value pairs assigned to the agent. These name/value pairs are used for agent selection when the test run configuration has agent constraints specified.
Attribute Name: The name of the attribute you want to use for agent selection, for example, ‘OS’. Refer to ‘Agent Selection’ for more context.
Attribute Value: The value for the given name, for example, “WindowsXP’.
Say you have Agent1 and you assign a number of name/value pairs in the grid:
Attribute Name Attribute Value
OS WindowsXP
SP SP1
Group A
Software MSOffice
Processor MULTICORE
Agent2 has the following attributes:
Attribute Name Attribute Value
OS WindowsXP
SP SP2
Group A
Processor SINGLECORE
Using the sample data from the ‘Agent Selection’ in the previous section, you can now select agents for a given test run.
Say you want agents that have ‘Processor’ attribute equal to ‘MULTICORE’:
Click on the ‘Test’ menu and click ‘Edit Test Run Configurations’ and select a run configuration. The test run configuration dialog will come up. Select ‘Controller and Agent’ in the left pane. In the agent properties grid, enter the following:
Name Value
Processor MULTICORE
When a test run is submitted Agent1 will match the agent constraints.
You want agents that have ‘OS’ attribute equal to ‘WindowsXP’
In the agent properties grid in the test run configuration dialog, enter the following:
Name Value
OS WindowsXP
Both Agent1 and Agent2 can be used for the test run since both match the agent constraints.
You want agents that have ‘OS’ attribute equal to ‘WindowsXP’ and ‘GROUP’ attribute equal to ‘B’
In the agent properties grid in the test run configuration dialog, enter the following:
Name Value
OS WindowsXP
GROUP B
Neither agent has ‘GROUP’ attribute equal to ‘B’, both agents have ‘GROUP’ attribute equal to ‘A’. No agent is returned that can execute the test run and the test run is aborted.
Remove
Clicking ‘Remove’ removes an agent from the agent list. Note that if the agent is running, it will continue to try to connect to the controller (and fail). So in addition you will need to stop the agent service running on the agent.
Offline/Online
If an agent is marked offline the controller will not attempt to use the agent to execute a test run. You can use offline to perform maintenance on the agent or investigate test results.
Restart
The controller will attempt to restart the agent through the service control manager interface (unless the agent name has an ‘*’ next to it). Typically the service control manager will stop and restart the agent service. Note that this is the same interface that is used in the Computer Management administrative tool.
If the controller cannot contact the agent (firewall) or the controller service user does not have the rights to stop/start a service on the agent machine, or the agent name has an ‘*’ next to it, the controller will bypass the service control manager and call into the agent service to tell it to exit the process. If the agent was running as a service, the service controller manager will restart it. If the agent was running as a normal process (has a ‘*’ next to the name) and was started through the supplied script file, QTAgentServiceUI.bat, the process will be restarted.
It takes a few seconds for the agent to restart and reregister with the controller. Clicking the ‘Refresh’ button will refresh the agent states in the dialog box.
Refresh
The ‘Refresh’ button causes the controller to query the agents for their state. Ant state changes are reflected in the main dialog’s agent list.
Delete Temp Files
Test runs will leave temporary files on the controller and agent machines (these may be deployed files, temporary test files, code coverage data files, …). They are kept in case there is a need to reference them. It is recommended that the temporary files be deleted when it is known they are no longer needed.
Restart Rig
Restarts the controller service and the agents the controller manages.
UI Testing on an Agent
If you want to run a test that has a UI or interacts with the desktop or user settings, you will need to use QTAgentServiceUI on the agent instead of QTAgentService.exe. QTAgentServiceUI works exactly like QTAgentService, except that it runs as a user process rather than a service. We recommend you configure your agent to automatically logon as a test user, and then in the startup for that user run QTAgentServiceUI.bat. This bat file simply runs QTAgentServiceUI over and over, so that if the agent service dies it will be automatically restarted.
Conclusion
The combination of Visual Studio Team Test Edition 2005 and Microsoft Visual Studio Team Load Agent 2005 allow for a flexible and powerful approach to testing. The combination allows you to execute tests on the local machine or with a pool of agents managed by a controller. You can create custom environments on agent machines and execute test runs in those environments using agent selection. You can control the load of each agent through weighting and simulate many browser requests from different locations using IP Switching. You can deploy custom data to agent machines and return custom data utilizing Deployment and Reverse Deployment. Through scripting you can further modify an agent environment and revert it to its original state when a test completes.
Visual Studio Team Test Edition 2005 and Microsoft Visual Studio Team Load Agent 2005 give you the tools and flexibility you need to effectively test in today’s complex environments.
Comments
- Anonymous
February 08, 2006
Check out this article:
Distributed Functional Testing with Visual Studio 2005 Team System - Anonymous
September 15, 2006
You've written your web application and tested it to ensure it all works but how do you know it will...