VSTS 2010 Feature: API for Processing Web Test Results
One request which we have received a number of times is something like, “The Web test playback UI is great for when you are in VS, but I need to share these results with others. Can I generate a report from the web test result?” In VS 2005 and 2008, the web test result was stored in a trx file which was an XML file. We did not have a public API for processing the web test results. In VS 2010, we have made some changes in this area.
First, the web test results are no longer completely stored in the trx file. The trx file has reference to a .webtestresult file which has the actual result. Look under you TestResults directory which is located in your TestProject directory by default. In this directory, you will see a number of trx files. If you then search into the folder that corresponds with the trx and then navigate into the IN directory, you will find the .webtestresult file. This file will have the name of the web test that was being executed. So if you ran WebTest1.webtest, you will find a result file names WebTest1.webtestresult.
We have made the API for processing .webtestresult files public. So now you can process this file and create your own custom reports for web test results. I am going to walk you through a demo for how to do this. We will create a simple console app that will read in the webtestresult file and then create an XML document. This will show you how to process the result file so you can generate your own custom reports.
WebTestResult Model
There are a few classes that you will have to become familiar with. These are all in the Microsoft.Visual Studio.TestTools.WebTesting namespace.
- WebTestResultDetailsSerializer – This class will be used to read the .webtestresult file.
- WebTestResultDetails – This class is what the serializer will return. This contains the full result. It contains a collection of iterations. Each iteration contains a collection of WebTestResultUnit objects.
- WebTestResultUnit - This is the base class for page, transaction, comments, loops, and conditional results.
- WebTestResultPage – This contains information about a page including the request, response, dependent Urls, redirected Urls, timing info etc.
- WebTestResultTransaction – This contains information about a transaction such as name and response time. It also contains a collection of WebTestResultUnit objects which are contained within the transaction.
- WebTestResultCondition – This contains information about a web test condition such as condition being tested and whether or not the condition was met. It also contains a collection of WebTestResultUnit objects which are contained within the condition.
- WebTestResultLoop – This contains information about a web test loop such as the loop condition. It contains a collection of WebTestResultLoopIteration objects. It has one object for each iteration of the loop.
- WebTestResultLoopIteration – This contains information about a particular loop iteration. It also contains a collection of WebTestResultUnit objects which are contained within the loop iteration.
- WebTestResultComment – This contains information about a web test comment.
Sample Console Application
Now that we have gone over the basic classes we will be using, let’s create a sample app which will create a custom report.
- First launch VS and create a new Console Application project.
- Right click on the project in solution explorer and select properties.
- Make sure the target framework is set to .NET Framework 4 and not .NET Framework 4 client profile
- Right reference node and select Add Reference…
- In the Add Reference dialog select the Microsoft.Visual Studio.QualityTools.WebTestFramework.dll
- Right click the project node and add a new class called ReportGenerator. Here is the code for that class. Let’s go over what is happening.
- GenerateReport – This method takes the webtestresult file. It creates and instance of the serializer which will then return the WebTestResultDetails. My sample is going to return an xml document, so I am also creating an Xml document here which I will be building up. Then you can see that it starts iterating over the Iterations. For each iteration it then iterates over the children in the iteration which are all WebTestResultUnit objects. It passes the WebTestResultUnit to ProcessResultUnit
- ProcessResultUnit – This method needs to figure out what kind of result unit it is and then do the appropriate thing. For example, for pages, it will Create a page node with the url and response times as attributes. Then it will iterates through the redirected requests and the dependent requests for the page. For a transaction, it will create a transaction node with the name and response time of the transaction. It will then iterate over the children of the transaction which are all WebTestResultUnit objects and call ProcessResultUnit for each of the children
using System.Xml;
using Microsoft.Visual Studio.TestTools.WebTesting;
namespace ConsoleApplication1
{
internal class ReportGenerator
{
public ReportGenerator()
{
}
public XmlDocument GenerateReport(string file)
{
//create the serialzer
WebTestResultDetailsSerializer serializer = new WebTestResultDetailsSerializer();
//deserialize the webtest result
WebTestResultDetails details = serializer.Deserialize(file);
//create root element for the document
XmlDocument doc = new XmlDocument();
XmlElement elem = doc.CreateElement("WebTestResultDetails");
doc.AppendChild(elem);
//now we need to loop through each iteration
foreach (WebTestResultIteration iteration in details.WebTestIterations)
{
XmlElement iterationElement = doc.CreateElement("Iteration");
elem.AppendChild(iterationElement);
//loop through and process each child. A Child could be a
//transaction, page, comment, innertest, etc.
foreach (WebTestResultUnit unit in iteration.Children)
{
ProcessResultUnit(unit, doc, iterationElement);
}
}
return doc;
}
private void ProcessResultUnit(WebTestResultUnit unit, XmlDocument doc, XmlElement elem)
{
if (unit is WebTestResultPage)
{
XmlElement pageElement = doc.CreateElement("Page");
elem.AppendChild(pageElement);
WebTestResultPage page = unit as WebTestResultPage;
pageElement.SetAttribute("Url", page.RequestResult.Request.UrlWithQueryString);
pageElement.SetAttribute("ResponseTime", page.RequestResult.Response.Statistics.MillisecondsToLastByte.ToString());
//process redirects
if (page.RedirectedPages.Count > 0)
{
//create the Redirects element
XmlElement redirectsElement = doc.CreateElement("Redirects");
pageElement.AppendChild(redirectsElement);
foreach (WebTestResultPage redirect in page.RedirectedPages)
{
ProcessResultUnit(redirect, doc, redirectsElement);
}
}
//process each dependent
if (page.RequestResult.DependantResults.Count > 0)
{
//create the DependentRequests element
XmlElement dependentsElement = doc.CreateElement("DependentRequests");
pageElement.AppendChild(dependentsElement);
foreach (WebTestRequestResult request in page.RequestResult.DependantResults)
{
XmlElement dependentElement = doc.CreateElement("DependentRequest");
dependentsElement.AppendChild(dependentElement);
dependentElement.SetAttribute("Url", request.Request.UrlWithQueryString);
}
}
}
else if (unit is WebTestResultTransaction)
{
WebTestResultTransaction transaction = unit as WebTestResultTransaction;
//create the transaction element
XmlElement transactionElement = doc.CreateElement("Transaction");
elem.AppendChild(transactionElement);
transactionElement.SetAttribute("Name", transaction.Name);
transactionElement.SetAttribute("IsIncludedTest", transaction.IsIncludedTest.ToString());
transactionElement.SetAttribute("ResponseTime", transaction.ResponseTime.ToString());
//now we need to process each child of the transacction which can any resultunit type.
//so iterate through children and call ProcessResultUnit
if (transaction.Children.Count > 0)
{
XmlElement children = doc.CreateElement("Children");
transactionElement.AppendChild(children);
foreach (WebTestResultUnit child in transaction.Children)
{
ProcessResultUnit(child,doc,children);
}
}
}
else if (unit is WebTestResultCondition)
{
WebTestResultCondition condition = unit as WebTestResultCondition;
//create the Condition element
XmlElement conditionElement = doc.CreateElement("Condition");
elem.AppendChild(conditionElement);
conditionElement.SetAttribute("Name", condition.ConditionStringRepresentation);
//now we need to process each child of the condition which can any resultunit type.
//so iterate through children and call ProcessResultUnit
if (condition.Children.Count > 0)
{
XmlElement children = doc.CreateElement("Children");
conditionElement.AppendChild(children);
foreach (WebTestResultUnit child in condition.Children)
{
ProcessResultUnit(child, doc, children);
}
}
}
else if (unit is WebTestResultLoop)
{
WebTestResultLoop loop = unit as WebTestResultLoop;
//create the Loop element
XmlElement loopElement = doc.CreateElement("Loop");
elem.AppendChild(loopElement);
loopElement.SetAttribute("Name", loop.LoopStringRepresentation);
//now we need to process each child of the loop which can any resultunit type.
//so iterate through children and call ProcessResultUnit
if (loop.Children.Count > 0)
{
XmlElement children = doc.CreateElement("LoopIterations");
loopElement.AppendChild(children);
foreach (WebTestResultUnit child in loop.Children)
{
ProcessResultUnit(child, doc, children);
}
}
}
else if (unit is WebTestResultLoopIteration)
{
WebTestResultLoopIteration loopIteration = unit as WebTestResultLoopIteration;
//create the LoopIteration element
XmlElement loopIterationElement = doc.CreateElement("LoopIteration");
elem.AppendChild(loopIterationElement);
loopIterationElement.SetAttribute("Number", loopIteration.IterationNumber.ToString());
loopIterationElement.SetAttribute("IsConditionalRuleMet", loopIteration.IsConditionalRuleMet().ToString());
//now we need to process each child of the loop which can any resultunit type.
//so iterate through children and call ProcessResultUnit
if (loopIteration.Children.Count > 0)
{
XmlElement children = doc.CreateElement("Children");
loopIterationElement.AppendChild(children);
foreach (WebTestResultUnit child in loopIteration.Children)
{
ProcessResultUnit(child,doc,children);
}
}
}
else if (unit is WebTestResultComment)
{
WebTestResultComment comment = unit as WebTestResultComment;
//create the Comment element
XmlElement commentElement = doc.CreateElement("Comment");
elem.AppendChild(commentElement);
commentElement.SetAttribute("Text", comment.Comment);
}
}
}
}
- Now go back to Program.cs file. This class just needs to know where the results file is and where to store the output file. So we will check to make sure we get this information and then pass it along the report generator. This class looks like this:
using System;
using System.Xml;
using System.IO;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
if (args.Length != 2)
{
Console.WriteLine("Usage: ReportGenerator <WebTestResultFile> <OutputFile>");
return;
}
if (!File.Exists(args[0]))
{
Console.WriteLine("WebTestResult file does not exist");
return;
}
if (File.Exists(args[1]))
{
Console.WriteLine("Output file already exists. Please specify a new name.");
return;
}
ReportGenerator generator = new ReportGenerator();
XmlDocument report = generator.GenerateReport(args[0]);
XmlTextWriter writer = new XmlTextWriter(args[1], null);
writer.Formatting = Formatting.Indented;
report.Save(writer);
}
}
}
Now compile your code. to run it, open a command prompt and go to directory that the exe was compiled into. You would run it with something like: ReportGenerator c:\Webtest1.webtestresult c:\NewReport.xml
Here is a sample output created from one of my webtests:
<?xml version="1.0"?>
<WebTestResultDetails>
<Iteration>
<Transaction Name="Transaction1" IsIncludedTest="False" ResponseTime="88.384">
<Children>
<Transaction Name="WebTest8.Storecsvs" IsIncludedTest="True" ResponseTime="56.825">
<Children>
<Page Url="https://sampleserver/Storecsvs" ResponseTime="28829">
<Redirects>
<Page Url="https://sampleserver/Storecsvs/" ResponseTime="27996">
<DependentRequests>
<DependentRequest Url="https://sampleserver/Storecsvs/IBuySpy.css" />
<DependentRequest Url="https://sampleserver/Storecsvs/images/grid_background.gif" />
</DependentRequests>
</Page>
</Redirects>
</Page>
</Children>
</Transaction>
<Page Url="https://sampleserver/Storecsvs/productslist.aspx?CategoryID=15&selection=1" ResponseTime="27737">
<DependentRequests>
<DependentRequest Url="https://sampleserver/Storecsvs/images/sitebkgrdnogray.gif" />
<DependentRequest Url="https://sampleserver/Storecsvs/IBuySpy.css" />
</DependentRequests>
</Page>
<Page Url="https://sampleserver/Storecsvs/ProductDetails.aspx?productID=394" ResponseTime="316">
<DependentRequests>
<DependentRequest Url="https://sampleserver/Storecsvs/IBuySpy.css" />
<DependentRequest Url="https://sampleserver/Storecsvs/images/sitebkgrd.gif" />
</DependentRequests>
</Page>
</Children>
</Transaction>
<Comment Text="Sample Comment" />
<Page Url="https://sampleserver/Storecsvs/AddToCart.aspx?ProductID=394" ResponseTime="251">
<Redirects>
<Page Url="https://sampleserver/Storecsvs/ShoppingCart.aspx" ResponseTime="0">
<DependentRequests>
<DependentRequest Url="https://sampleserver/Storecsvs/images/sitebkgrd.gif" />
<DependentRequest Url="https://sampleserver/Storecsvs/IBuySpy.css" />
</DependentRequests>
</Page>
</Redirects>
</Page>
<Loop Name="Loop ( Repeat 1 times )">
<LoopIterations>
<LoopIteration Number="1" IsConditionalRuleMet="True">
<Children>
<Page Url="https://sampleserver/Storecsvs/ShoppingCart.aspx" ResponseTime="27729">
<DependentRequests>
<DependentRequest Url="https://sampleserver/Storecsvs/IBuySpy.css" />
<DependentRequest Url="https://sampleserver/Storecsvs/images/sitebkgrd.gif" />
</DependentRequests>
</Page>
</Children>
</LoopIteration>
<LoopIteration Number="2" IsConditionalRuleMet="False" />
</LoopIterations>
</Loop>
<Page Url="https://sampleserver/Storecsvs/ShoppingCart.aspx" ResponseTime="542">
<DependentRequests>
<DependentRequest Url="https://sampleserver/Storecsvs/images/sitebkgrd.gif" />
<DependentRequest Url="https://sampleserver/Storecsvs/IBuySpy.css" />
</DependentRequests>
</Page>
</Iteration>
</WebTestResultDetails>
I hope this helps getting you started in creating your own custom web test result reports. There is plenty of other information available to you. My sample just gives you an idea of how to parse through the objects. More or less anything that you see in the web test playback UI is available in the result objects.
Comments
Anonymous
July 20, 2012
I compiled this and tried to get it working. But this is the runtime error I get, which seems to say the file format has changed..... Unhandled Exception: Microsoft.VisualStudio.TestTools.WebTesting.WebTestException: Failed to serialize WebTestResultDetails. ---> System.Runtime.Serialization.SerializationException: The input stream is not a valid binary format. The starting contents (in bytes) are: EF-BB-BF-3C-3F-78-6D-6C-20-76-65-72-73-69-6F-6E-3D ... at System.Runtime.Serialization.Formatters.Binary.SerializationHeaderRecord.Read(__BinaryParser input) at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadSerializationHeaderRecord() at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run() at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) at Microsoft.VisualStudio.TestTools.WebTesting.WebTestResultDetailsSerializer.Deserialize(String filePath) --- End of inner exception stack trace --- at Microsoft.VisualStudio.TestTools.WebTesting.WebTestResultDetailsSerializer.Deserialize(String filePath) at ConsoleApplication1.ReportGenerator.GenerateReport(String file) at WebTestSummarizer.Program.Main(String[] args)Anonymous
July 20, 2012
Figured it out just after I posted this.....Anonymous
December 13, 2012
how i get the values of context parameters?Anonymous
May 14, 2013
Brilliant, exactly what i was looking for..Anonymous
October 27, 2013
Hi Ken I get the same error, what did you figure out? Anybody else know how to fix this?