What's a web test?
Since not too many people have played with the first community tech preview release of Visual Studio Team System's web testing (codenamed Ocracoke), I thought I'd share some basics about web tests. Fundamentally, a web test is just a list of requests that simulate a user's interaction with a web application. Some web tests may be designed to stress a web app by being run under heavy load and other web tests may thoroughly validate that the web app's features are working correctly.
A web test will usually be created by using the browser recorder to record a series of requests. Below, you can see a very simple web test I recorded:
This web test is about as basic as it gets. I browsed around the ASP.NET commerce starter kit (aka IBuySpy) and added a product to my shopping cart. This web test could be run individually to make sure none of the requests result in server errors or as part of a load test to look for any performance problems. While this level of testing may be sufficient, you could go one step further and add a validation rule that actually verifies the shopping cart contains the product I selected. You could also databind the ProductID parameter to a database so that each virtual user selects a different product.
Web tests like the one shown above can handle a large chunk of testing scenarios, but sometimes you need more control. Coded web tests give you the ability to using branching and looping constructs, perform custom databinding, and much more. Since coded web tests are written in a .NET language, you can make use of the entire .NET framework as well as any other code. I'll cover coded web tests in more depth in a future post, but here's the C# code generated by converting the web test shown above to a coded web test:
public override IEnumerator<WebTestRequest> GetRequests()
{
WebTestRequest request1 = new WebTestRequest("https://www.asp.net/CommerceStarterKit/default.aspx");
request1.ThinkTime = 6;
yield return request1;
WebTestRequest request2 = new WebTestRequest("https://www.asp.net/CommerceStarterKit/productslist.aspx");
request2.ThinkTime = 4;
request2.QueryStringParameters.Add("CategoryID", "15");
request2.QueryStringParameters.Add("selection", "1");
yield return request2;
WebTestRequest request3 = new WebTestRequest ("https://www.asp.net/CommerceStarterKit/productslist.aspx");
request3.ThinkTime = 2;
request3.QueryStringParameters.Add("CategoryID", "17");
request3.QueryStringParameters.Add("selection", "4");
yield return request3;
WebTestRequest request4 = new WebTestRequest("https://www.asp.net/CommerceStarterKit/ProductDetails.aspx");
request4.ThinkTime = 9;
request4.QueryStringParameters.Add("productID", "378");
yield return request4;
WebTestRequest request5 = new WebTestRequest("https://www.asp.net/CommerceStarterKit/AddToCart.aspx");
request5.ThinkTime = 10;
request5.QueryStringParameters.Add("ProductID", "378");
yield return request5;
}
As you can see, the object model closely matches the web test tree view you saw in the screenshot. That's because non-coded web tests use the exact same object model internally.
What do you think? Any questions or comments about web tests, coded web tests, or Ocracoke in general?
Comments
- Anonymous
August 23, 2004
Does this webtest handle Https connections properly - this was a big problem in the Application Center test environment... - Anonymous
August 23, 2004
Scott,
Yes, we fully support HTTPS connections. There is no difference between recording and running a web test that uses SSL and one that doesn't. - Anonymous
August 23, 2004
Hi Scott,
The web testing is main pain for my team. ACT is almost useless and I was waiting for a good test system that would support .NET Framework.
He developed a few internal applications, but still they are not perfect and very limited.
The biggest problem I have faced is the authentication and cookie support.
ACT does not support forms authentication mechanism. Will Ocracoke be able to work with Forms Authentication?
I have tried to write a complicated ACT script that would do work with forms authentication and viewstate. It is possible by managing viewstate manually. Unfortunately I ran into cookies problem. Cookie: (automatic) would not work every time I ran the script. It was working 6 out of 10 times. That means I can not count on that script and ACT in general. - Anonymous
August 23, 2004
Alex,
Ocracoke does support Forms Authentication and also has a feature to automatically track viewstate (as well as other hidden fields) for you. You shouldn't need to ever manage viewstate manually. We can also automatically track session cookies. Forms authentication will be recorded by our browser recorder and you can then replace the recorded username and password with databound values if you want to simulate different users logging in. - Anonymous
August 23, 2004
Is there any support for testing web services directly (as opposed to, say, fetching web pages that invoke web services)? - Anonymous
August 24, 2004
Kevin,
you can use AmberPoint to stress test web service. It works with Framework 1.1.
here is a link: http://express.amberpoint.com/expressonline/users/Home.xhtml?apdotcom=weblink
AmberPoint will be included in VS 2005.
Scott or Josh :)
Is the Orcacoke included in VS 2005 Beta 1?
I dont recall TeamSystem included in that build, but I did not try it, so I may be wrong. - Anonymous
August 24, 2004
The comment has been removed - Anonymous
August 24, 2004
Josh,
How about "Visual Studio 2005 Community Technology Preview May 2004 - 32 Bit (English)" that I can download from MSDN as Universal MSDN Subscriber? - Anonymous
August 24, 2004
Alex,
The May CTP release does include Team System, but it is built on a pre-Beta 1 version of VS 2005. There are several big bugs in that version of the .NET framework that can cause web load tests to die after a relatively short period of time. That release would be fine if you just want to play with Ocracoke to see what it's like, but, like I said, there's a newer CTP release due out very soon. - Anonymous
August 24, 2004
Thanks Josh,
I appreciate your help. - Anonymous
August 24, 2004
Ok, that is just too cool. I think you have given me the bump I need to get back into playing with Team System again.
Great tutorial! Thanks! - Anonymous
August 25, 2004
Josh,
Can you say how widely available the CTP will be? MSDN subscribers only, or public like the "Express" products? - Anonymous
August 25, 2004
Josh...I don’t have enough words to describe my feelings after I have installed CTP (May 2004) and tried Ocracoke. It is amazing, it is great...
"You are something" (c) Robert De Niro ”Analyze That. 2002”.
That is the best web test and load test tool I have seen so far. It has all features that I was looking for and a lot that I could not even imagine. Using Ocracoke reduces the number of internal tools and third party software needed to help my web team to develop better and stable applications, calculate the load of servers in the farm and etc.
I know you said that this May 2004 preview is not stable, but it is still looks and works very well. It works with Form Authentications, popup windows, I love the way I can setup the performance counters – just select the ones you want and then add web servers, simple as it is.
I’m setting a few dedicated machines just for the stress testing.
You did great work and I really appreciate that. Thanks a lot. - Anonymous
August 27, 2004
*I'm sorry for responding so late to the last two comments, but I didn't see them until now. Our blog system unexplainably stopped sending me notification emails.
Stuart,
The CTP release will be available online to MSDN subscribers. It will also be handed out at the VSLive! conference in Orlando.
Alex,
Wow! That's some awesome feedback and we're very happy you already like the product so much! I can't wait to hear your feedback after you see all the new features that have been added in our current coding milestone :) - Anonymous
August 30, 2004
Josh,
1)I have noticed that requests/seconds are slightly better when I use Ocracoke than the same tests but using ACT. Sometimes by 30-50 requests/seconds. Is that because the calculations are not correct and it will be fix later in Ocracoke or it is because ACT is wrong?
2) Is there a way to display the results of two or more tests in Ocracoke? In ACT you can select the result and they will be displayed on one graph, so it is very very useful. - Anonymous
September 01, 2004
Alex,
1) There are many significant differences between the ACT engine and Ocracoke's engine so I would not expect the requests/second numbers be exactly the same. BTW, are you using coded tests in Ocracoke and/or ACT?
2) I believe the build you have doesn't include a way to view load test results side by side (other than opening two result windows in VS), but reporting features have been added in the current milestone. One of the included reports will let you compare runs. - Anonymous
September 01, 2004
Josh,
I just installed August version of CTP. I will re-test my applications to see if there is a deference.
About my question #1: I have used coded test in ACT and recorded test in Ocracoke.
I understand that recorded test loads images as well, when coded version for ACT does not load anything but ASPX pages, but I have tested empty html pages as well and ACT displayed lower numbers.
Anyway, I'm testing this new version and leave a feedback for you :)