通过

Guide to get started with Visual Studio Web Load Testing and Automation

  • 项目

Applies to VS 2015 but other versions mentioned.

This article provides consolidated information in a concise presentation for anyone who wants to start a web load test project using Visual Studio. You may have noticed that there is plenty of information on MSDN and visualstudio.com that tells you about cloud testing/on premises requirements, agents, controllers, webtests, plugins, etc.

The amount of information can be easily overwhelming, this article can be used as a starting guide for newcomers to understand the basic setup and to find the best route into achieving the results desired.

1. Software Requirements. Versions and licenses

Visual Studio

The first thing you will need is Visual Studio. The Web Load & Performance Testing features of Visual studio are only available on versions: Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013.

Other type of tests such as Coded UI, Manual tests and Unit testing are also available on Visual Studio Test Professional 2015 however these are all functional tests. We'll focus on performance web tests for now, this is NOT available on VS Test Professional. You can find a feature comparison among VS versions here: Compare Visual Studio 2015 Offerings.

Virtual User Licenses

On premises. No virtual user licenses are needed. You can execute load tests with any number of virtual users (as far as your hardware resources allow you) as long as you have any of the following:

- Visual Studio Enterprise 2015 with MSDN

- Visual Studio Enterprise 2015 annual and monthly subscribers (check Visual Studio 2015 Licensing White Paper)

- Visual Studio Ultimate 2013 with MSDN (check Visual Studio 2013 Licensing White Paper)

* In Visual Studio Enterprise (or Ultimate) trial version, the virtual user count is limited to 250.

Cloud-Based testing. No extra software installation needed. You do need:

- A Visual Studio Online (VSO) account (get one here).

- That gets you 20,000 virtual user minutes every month to load test at no extra charge (check how virtual users minutes work and test duration limitations here: VSO Virtual users minutes).

Controller and Agents Software

On premises. If you are planning to load test an application in an on-premises environment, you will need a Controller and at least 1 agent machine (check next section for hardware requirements). You can find the installable software for these two components based on your Visual studio version here:

- Download Agents for Microsoft Visual Studio 2015  - Update: Don't use the Test Agents for VS 2015 for load testing. It is meant for continuous tests on build scenarios, it doesn't have a test controller or a configuration tool. Instead use VS Test Agents 2013 Update 5:

- Download Agents for Microsoft Visual Studio 2013 Update 5

 Cloud-Based Load testing. No additional software installation needed.

 

2. Infrastructure/Hardware requirements

Visual Studio (IDE) System Requirements

Visual Studio is used to record webtests, kick off Load Tests and visualize results. It is not recommended to install it on the Controller machine or Test Agents. The system requirements for the machine to host either Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013 are:

Controller and Agents Hardware Requirements

On premises. If you are planning a load test project in a on premises environment, you will need a controller and at least 1 test agent, to set this up you will need the following:

  • OS/Framework requirements
    • Controller: Windows 7 SP1 / 8 / 8.1 or Windows Server 2008 R2 SP1 / 2012 / 2012 R2
    • Test Agents: Windows XP SP3 / 7 SP1 / 8 / 8.1 or Windows Server 2008 R2 SP1 / 2012 / 2012 R2.
    • Both: .NET Framework 4.5
    • More information here: System requirements for Controller and Agents

You can also execute local test runs (without controller or test agents) directly from your Visual Studio host, this is done normally for debugging purposes since you are limited by resources.

  • Hardware sizing

The table below shows the recommended hardware requirements. You can use this table to plan how many test agents and what size of controller suit your load testing needs:

Configuration Component CPU HD Memory
< 500 virtual users Test agent 2.6 GHz 10 GB 2 GB
< 1000 virtual users Test agent Dual processor 2.6 GHz 10 GB 2 GB
N x 1000 virtual users Test agent Scale out to N agents each with Dual 2.6 Ghz 10 GB 2 GB
< 30 computers in the test environment. This includes agents and servers under test. Test Controller 2.6 GHz Read load test repository section
N x 30 computers in the test environment. This includes agents and servers under test. Test Controller N 2.6 GHz processors Read load test repository section

 

  • Load Test Repository

Load tests results may be stored in the Load Test Results Repository, which is a SQL database. The Results Repository database is created by setup for controllers (or created automatically on the first local run of a load test), a the database will be created automatically if the load test schema is not present.

SQL Server 2012 Express LocalDB, which is installed with Visual Studio is the default database server for load tests. If SQL Server Express is detected with an existing load test database, Visual Studio Enterprise will try to connect to it and use it.

However, for heavier database needs you should consider upgrading to a full SQL Server to provide further scaling potential. Also, SQL Express is limited to using a maximum of 4 GB of disk space. If you will run many load tests over a long period of time, you should consider configuring the load test results store to use an instance of the full SQL Server product if available. More on Load Test results repository.

Note: A single 1 hour load test can take from 100MB to 500MB depending on the number of webtests, performance counters, VMs you are monitoring, etc. Plan storage needs accordingly and take in consideration that this information is temporarily stored in the Test Agents, collected in the Controllers RAM memory before finally being placed in the SQL Repository. Plan RAM Memory needs accordingly.

  • Alternative : Testing using VSO or TFS instead of Controller

For test scenarios (still on premises) using Visual Studio Online (VSO) or Team Foundation Server (TFS) 2015, you won't need a test controller because Agents for Microsoft Visual Studio 2015 handle orchestration by communicating with VSO or TFS 2015. For example, you're running automated tests with your build and release workflows in VSO or TFS 2015.

Cloud-Based Load testing

You can use cloud-based load testing to avoid having to use your own resources and machines. In this case you don't need a Controller or Test Agents. The cloud-service provides virtual machines in the cloud that generate the load needed to test your target application/website.

All you need is a Visual Studio Online (VSO) account. Check 'Getting started with Load Tests' further down in this document.

 

3. Getting started with Webtests

Basic components

At the most basic level you will need to do the following steps to get started:

  • Create a web performance and load test project
  • Record a web performance test.
  • Create a load test.
  • Run and analyze your load test

 Follow this article to learn how to create the most basic components.

Validation

Validation rules help verify that a Web application is working correctly by validating the existence of text, tags, or attributes on the page returned by a Web request. Validation rules can also verify the time that it takes a request to finish, and the existence of form fields and their values.

Check this article to find out how to add predefined Validation rules to your webtest. You can also create your own Validation rules, check Customization section further down on this page.

Parametrization

o    Constant parameters. You can create a Context Parameter for any value in a request of a WebTest. The constant parameters are displayed at the bottom of your webtest. A simple case where you would want to do this is for host or website URLs. Read this article for more information.

o    Dynamic parameters. Web sites and applications use dynamic values, this is a value that is generated every time that a user runs the application. A dynamic parameter can cause your Web performance test playback to fail because it has a different value every the test is run. Therefore, you cannot play back recorded values. (E.g. a session ID). To solve this situation you have a couple of options (follow link to articles):

- Auto-detect dynamic values and promote them to WebTest parameters. Use VS auto-detect feature to identify and parameterize dynamic values (it may not catch all values).

- Add an Extraction Rule. Find dynamic values by using 'quick find' on recorded results then add an extraction rule:

      • Extraction rules extract data from the responses to Web requests and store the results in the test context as name/value pairs. Extraction rules are predefined in Visual Studio (you can also create your own, check Customization section), you can use them to extract form fields, text, attributes, headers, regular expressions, and hidden fields, Check full article here.

Data Binding

 You can bind parameters (e.g. form post parameters) on the requests of your WebTest to data sources to provide different values to the same test. This makes your load test more solid and realistic. You have several options for the data source type:

- SQL database (several versions permitted, check FAQ)

- CSV text files

- XML files (or SOAP XMLs)

 Check this article for a step by step guide.

 

4. Getting started with Load Tests

Load Test setup

Load Test can be created in a Load Test projects. If you followed the article at the start of Section 3 of this page you must already have a simple Load test. Some of the Load test settings and scenarios can be setup on the initial Wizard or directly on the load test afterwards. Below are descriptions for the main sections:

Scenarios

Load tests contain scenarios, which contain Web performance tests. Scenarios are important because they give you flexibility in configuring test characteristics that allow for simulation of complex, realistic workloads. You can have more than one scenario per load test.

 Use the Load pattern, test mix model and test mix to specify what webtests to use, the manner in which the load will be applied to your application and pacing.

 You can also use Browser mix, Network mix to make your workload more realistic. Check MSDN article on editing Load tests scenarios.

Counter Sets, Counters and Mappings

Counter sets that are useful when you analyze performance counter data. The counter sets are organized by technology such as Application, .NET Application, IIS, or SQL. You can create counter sets during the initial New Load Test Wizard or later on the load test file itself.

There is a set of predefined counter sets or you can create your own. You can also add counters to the default Counter Sets or to your own. The counters are added from Perfmon.

 On the Run settings section of a load test you can specify Counter Set mappings where you can add computers (by DNS or IP) and map them to Counter sets. Visual studio will collect data for every counter on a set for each computer where it is mapped to.

 To learn more about Counter Sets, thresholds, sampling and other tasks, check this article on MSDN.

Run Settings

Run settings are properties that organize the manner in which a load test runs. You can have more than one run setting in a load test but only one can be active at a time while the other settings provide a quick way to select an alternative settings.

Settings are used to specify things like test duration, Logging, SQL server collection, warm up duration, sampling rate or validation setting between others. You can also add Counter Set Mappings to map computers to Counter Sets.

Check this article for full description of tasks available.

Running Load Tests On Premises

Test Settings file

Make sure your Load test is configure to run On premises by checking the General section of your active .testsetting file on your Solution. Test Settings are not the same as Run Settings.

 Other items to configure on this window are Roles and Timeouts. On the Roles section, test execution method should be set to 'Remote execution' if you want to use your Controller and Test Agents. Check this couple of article to learn more about Test settings: Managing Test Controllers and Test Agents with Visual Studio, Edit Test Settings.

Controller-Agents connectivity

Also make sure the Controller can communicate with the Test Agents by checking the Load Test > Manage Test Controllers. That dialog should show if the controller has any Test Agents connected to it. More information here: Manage Test Controllers.

Running the Load Test

From that point forward you should be ready to run a Load Test. During or after run time you have access to Graphs and Tables with throughput, response times information, performance counters for the computers you configured and details on vusers activity. Check this tutorial to learn more: Run and analyze your load test.

Running Load Tests using the Cloud Service

There are a couple of options for running a Load test with the Cloud Service:

  • You can run a Load test from Visual Studio (Enterprise 2015 or Ultimate 2013) and Visual Studio Team Services.
    • As mentioned above you will need a VS Team Services account.
    • Your solution needs to be connect to TFS.
    • In your solution; if you open your active .testsettings file, on the General section select 'Run tests using Visual Studio Online':
    • VS_1
    • From there you can specify the Location you want your load to come from on the Load Test file and run the Load test. Check the full tutorial here, specifically the 'Run and analyze your load test' section.
  • Your second alternative is to run a basic (and much more limited) load test directly in Visual Studio Team Services. The up side is that you don't need a load test project (or Visual Studio). You do need a Visual Studio Enterprise monthly or annual subscription or a MSDN subscription.
    • To do this you need to go to the Load Test Hub on your Visual Studio Team Services account page.
    • Setup is simple, just specify URL, Load location and some simple Test settings. Check the com article here.

 

5. Customization

Visual Studio offers extendibility options to increase the built-in functionality. Follow the articles below to learn more about the different options:

- Custom Validation Rules. You can create your own validation rules. To do this, you derive your own rule class from a validation rule class. Validation rules derive from the ValidationRule base class.

- Custom Extraction Rules. You can create your own extraction rules. To do this, you derive your own rules from an extraction rule class. Extraction rules derive from the ExtractionRule base class.

- Custom Plug-in's. You can create custom plug-in to write code and attach it to a load test or a Web performance test. You can use the load test API and the Web Performance test API to create custom plug-ins for tests to expand to the built-in functionality.

FAQ

How many vusers can I run per agent?

As mentioned above the recommendation is up to 500 for a 2.6 GHz processor and up to 1000 for a Dual 2.6 GHz processor. This can change depending on the length of the webtests, the payloads of the application responses and the use of custom plug-ins.

What version of SQL do I need?

SQL Express or SQL Express 2012 Localdb for local test runs and debugging. Full SQL for heavier needs such as an Enterprise level application performance test project, for example the standard version of SQL Server 2014, 2012, or 2008 R2.

Can load tests use other test types in their test mix besides web performance tests?

Yes, you can include unit tests and coded UI tests.

Can virtual users simulate pausing between test steps?

Yes, you can specify think times to simulate the time spent by a user on a web page.

Can I mix versions of TFS, Microsoft Test Manager, the test controller, and test agent?

Yes, here are the compatible and supported combinations:

TFS Microsoft Test Manager, with Lab Center Controller Agent
2015: upgrade from 2013 2013 2013 2013
2015: new install 2013 2013 2013
2015: upgrade from 2015 or new install 2015 2013 2013
2013 2015 2013 2013

 https://msdn.microsoft.com/en-us/library/dd648127.aspx#Anchor_2

What firewall permissions do I need?

The default port used by the test controller is 6901 and the test agent's default port is 6910. The client (Visual Studio machine) uses a random port by default which is used to receive the test results from the test controller. For all incoming connections, the test controller authenticates the calling party and verifies that it belongs to specific security group. This is true regardless of the test rig configuration you are using: on premises or cloud based testing:

VS_TestAgents_Ports_Requirements

 

You should create exceptions for the mentioned ports on the firewall, you can also change the port. Check this msdn article for more information: Configuring Ports for Test Controllers and Test Agents

What databases can I use as a data source?

You can use the following:

  • Microsoft SQL Azure.
  • Any version of Microsoft SQL Server 2005 or later.
  • Microsoft SQL Server database file (including SQL Express).
  • Microsoft ODBC.
  • Microsoft Access file using the .NET Framework provider for OLE DB.
  • Oracle 7.3, 8i, 9i, or 10g.

What is a good place to learn more?

Aside from MSDN you can go to Channel 9 videos for a more visual learning experience:

https://blogs.msdn.microsoft.com/visualstudioalm/2015/12/10/visual-studio-2015-test-tools-getting-started-content/

Comments

  • Anonymous
    June 12, 2016
    Update: Don't use the Test Agents for VS 2015 for load testing. It is meant for continuous tests on build scenarios, it doesn't have a test controller or a configuration tool. Instead use VS Test Agents 2013 Update 5
  • Anonymous
    August 21, 2016
    great post! thanks!
  • Anonymous
    February 16, 2017
    How can I get to know about the license details of Load test tool in VS
    • Anonymous
      May 21, 2017
      Hi, thanks for posting a comment. If by "Load test tool" in VS you mean the Performance Web and Load Test features of Visual Studio, they do not have separate licensing once you get Visual Studio Enterprise. For VS Enterprise pricing, you can check this URL: https://www.visualstudio.com/vs/pricing/. You do not need virtual user licenses for on-premises load tests, you are limited by hardware only. For cloud-testing you do need user-minutes, you get 20,000 for any VSO account.
  • Anonymous
    June 15, 2017
    If I want to test using 250 more user load, Do I have to Visual studio 2017 ultimate?(or 2015 enterprise or etc) Or Do you any other method? For example, no purchase VS 2015 or 2017, but using only web site(Team Service).I want to only 250 more user load for load testing, other function not needed.