Explanation of “Off-Box Inclusive Testing” and how it impacts results and troubleshooting
Explanation of “Off-Box Inclusive testing” and how it impacts troubleshooting
An “Off-Box” request is any request that is made by a client to a machine that is not part of the core environment. As an example, if a user opens Internet Explorer and goes to the main Contoso.com website, Internet Explorer will go to the Contoso machines (the core environment). However the browser will also go to a few 3rd party sites on the Internet to retrieve images and possibly data. These “Off-Box” calls are made by Internet Explorer because the core environment told IE to go to this place to get the image or data.
Once the core environment tells IE where it should get the image or data, the core environment is no longer involved in any way with completing the final page in IE. The fact that these calls are made to external sources, which are shared by applications from other companies means we have no way of directly influencing, or even knowing what load or issues they might have that can change our numbers.
As an example, here is a simple test to the <Contoso> default web site. The request was made five times in a row and timing was measured from each one.
The results show:
· The performance improvement between runs 3 and 4 was:
o Total with OFF-BOX: 2.9 times longer (289%)
o Total of ON-BOX only: 6.4 times longer (642%)
· The performance improvement between runs 4 and 5 was:
o Total with OFF-BOX: 1.6 times longer (157%)
o Total ON-BOX only: 4 times faster (26%)
So test run 4 would give us a *false sense* that degradation was not nearly as bad as it really was. Likewise, test run 5 makes us believe that our machine was slower than in test run 4, when in fact the machine itself was 4 times faster.
Here are the details of the above results, showing that performance improvement and degradation measurements involving substantial “off-Box” calls can lead to erroneous interpretation of the results:
· The page I requested was https://www.contoso.com/Default.aspx
· By making that single request, my browser was instructed to make 34 more requests to sites such as:
o Images.buy-here.com
o Origin.buy-here.com
o Tracking.searchmarketing.com
· My harness measured the time for every request, including dependent requests.
Explanation of “Dependent Request exclusion” testing
Dependent Request Exclusion testing is similar to Off-Box testing, but it involves not making calls to static dependent files which can either be served by cache or return a 304 Not Modified. Even though you can sort requests in the result set and target specific ones, the TEST and PAGE averages reported will be substantially impacted.
When to use these methods
If you are doing tuning and testing, or application specific measurements, you should include the above methodologies. If, however, you want to get a rough idea of client viewed response time, you should not utilize either of these methods.
How to use these methods
If you want to exclude the types of requests listed above, you can use one of the below webtest plugins.
[DisplayName("TSL Toggle Dependent Requests")]
[Description("Enables or Disables all dependent requests in the web test")]
public class ToggleDependentRequests : WebTestPlugin
{
#region Properties
[DisplayName("Disable Dependent Requests")]
[Description("If true, dependent requests will NOT be executed")]
[DefaultValue(true)]
public bool bDisableDependentRequests { get; set; }
#endregion //Properties
public override void PreRequest(object sender, PreRequestEventArgs e)
{
if (bDisableDependentRequests)
e.Request.ParseDependentRequests = false;
}
}
[DisplayName("TSL Skip Calls To URL")]
[Description("Removes any requests that do meet a certain criteria WRT the URL of the request.")]
public class SkipAllCallsToUrl : WebTestPlugin
{
[DisplayName("Process Redirect Only")]
[Description("If true, this plugin will only execute on a redirect request. None of the main requests in the webtest will be affected.")]
[DefaultValue(false)]
public bool ProcessOnlyRedirection { get; set; }
[DisplayName("Plugin Enabled")]
[Description("If set to FALSE, the plugin will not perform any work")]
[DefaultValue(true)]
public bool bEnabled { get; set; }
[DisplayName("Prohibited Text")]
[Description("Excludes any request that contains this string value/or a context param. " +
"If this value is empty, the rule will remove any dependent requests with a different Base URL than the main request.")]
[DefaultValue("")]
public string RequestsURLCannotContain { get; set; }
public override void PreRequest(object sender, PreRequestEventArgs e)
{
if (bEnabled)
{
// NOTE: This routine is what handles context params passed in as values in the plugin
string cleanRequestsURLCannotContain = "";
if (RequestsURLCannotContain.Contains("{{"))
{
RequestsURLCannotContain = RequestsURLCannotContain.Replace("{{", "");
RequestsURLCannotContain = RequestsURLCannotContain.Replace("}}", "");
cleanRequestsURLCannotContain = e.WebTest.Context[RequestsURLCannotContain].ToString();
}
else
cleanRequestsURLCannotContain = RequestsURLCannotContain;
if (e.Request.Url.ToString().Contains(cleanRequestsURLCannotContain))
{
e.Instruction = WebTestExecutionInstruction.Skip;
}
}
}
}
UPDATE: New code from a teammate of mine
Robert George ("Offending the testing gods") shared this code with me which offers much more flexibility than the above code. So if you want to keep it simple, use the above. If you want capability, use the below.
//*********************************************************
// Copyright (c) Microsoft. All rights reserved.
// THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
// ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
// IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
// PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.
//
//*********************************************************
using System;
using System.Collections.Generic;
using Microsoft.VisualStudio.TestTools.WebTesting;
using Microsoft.VisualStudio.TestTools;
using System.ComponentModel;
using System.Diagnostics;
namespace MTSL
{
[Description("Utility plugin that supports various exclusion criteria for requests that you can define.")]
public class ExcludeRequestsPlugin : WebTestPlugin
{
//constructor
public ExcludeRequestsPlugin()
{
this.DependentMustContain = string.Empty;
this.DependentMustNotContain = string.Empty;
this.PrimaryMustContain = string.Empty;
this.PrimaryMustNotContain = string.Empty;
this.ListDelimiter = ";";
this.RemoveAllDependentRequests = false;
this.Enabled = true;
this.SuppressOutputComments = false;
this.IgnoreCase = false;
}
[Description("Allows you to supress comments output by this plugin by setting true/false")]
[DefaultValue(false)]
public bool SuppressOutputComments { get; set; }
[Description("Allows you to enable/disable this webtest plugin by setting true/false")]
[DefaultValue(true)]
public bool Enabled { get; set; }
[Description("Allows case insensitve comparisons by setting IgnoreCase=true. Note: Context Paramaters are always treated as case sensitive.")]
[DisplayName("Ignore Case")]
[DefaultValue(false)]
public bool IgnoreCase { get; set; }
[Description("Automatically removes all dependent requests for every request")]
[DefaultValue(false)]
public bool RemoveAllDependentRequests { get; set; }
[Description("Excludes any dependent request that does not contain this string value. Can use context param {{Paramxyz}} syntax also. Example: request -> https://99.1.1.10/images/img.jpg would not be processed if you specify https://10.100.100.10. This can also be a delimited List of values if you have more than one.")]
[DefaultValue("")]
[DisplayName("Dependents Must Contain")]
public string DependentMustContain { get; set; }
[Description("Excludes any dependent request that contains this string value/or a context param.In case of conflict, this property has higher precedence. For example: https://99.1.1.10/images/img.jpg would not be requested if the property was set to img.jpg. This is very useful for dealing with requests that are known to cause 404 responses. This can also be a delimited List of values if you have more than one.")]
[DefaultValue("")]
[DisplayName("Dependents Must Not Contain")]
public string DependentMustNotContain { get; set; }
[Description("Excludes any primary request that does not contain this string value. Can use context param {{Paramxyz}} syntax also. For example: https://99.1.1.10/images/img.jpg would not be processed if you specify https://10.100.100.10. This can also be a delimited List of values if you have more than one.")]
[DefaultValue("")]
[DisplayName("Primary Must Contain")]
public string PrimaryMustContain { get; set; }
[Description("Excludes any primary request that contains this string value. In case of conflict, this property has higher precedence. You may use context param {{Paramxyz}} syntax also. For example: https://99.1.1.10/images/img.jpg would not be processed if you specify img.jpg as the value for the property. This can also be a delimited List of values if you have more than one.")]
[DefaultValue("")]
[DisplayName("Primary Must Not Contain")]
public string PrimaryMustNotContain { get; set; }
[Description("Enter the delimiter that you would like to use to seperate list values in the other propeties in the plugin. For example, use a comma or semi-colon to seperate the ExcludeDependentRequests properties.")]
[DefaultValue(";")]
[DisplayName("List Delimiter")]
public string ListDelimiter { get; set; }
public override void PreRequest(object sender, PreRequestEventArgs e)
{
//call base class PreRequest
base.PreRequest(sender, e);
// only if the plugin is enabled we will continue
if (this.Enabled)
{
//if the request has already been set to skip, by another plugin for example, then there is no reason to
//execute the work in this plugin
if (e.Instruction == WebTestExecutionInstruction.Skip)
{
if (!this.SuppressOutputComments)
{
e.WebTest.AddCommentToResult(this.ToString() + " -> request instruction already set to Skip. No processing by this plugin occured.");
Debug.WriteLine(this.ToString() + " -> request instruction already set to Skip. No processing by this plugin occured.");
}
return;
}
//if they dont want any dependents we just set the request ParseDependentRequests = false
if (this.RemoveAllDependentRequests)
{
e.Request.ParseDependentRequests = false;
if (!this.SuppressOutputComments)
{
e.WebTest.AddCommentToResult(this.ToString() + " -> set ParseDependentRequests to FALSE, no dependent requests will be executed.");
Debug.WriteLine(this.ToString() + " -> set ParseDependentRequests to FALSE, no dependent requests will be executed.");
}
}
//our private string vars to work with
string mustContain = this.PrimaryMustContain;
string mustNotContain = this.PrimaryMustNotContain;
//If they are using a context param, we need to fix this up for them.
//We know the context param format because it contains the "{{" as an example-> {{Parameter1}}.
if (mustContain.Contains("{{") || mustNotContain.Contains("{{")) //this is just a short circuit if we have no processing needed
{
//make sure the context param exists
if (e.WebTest.Context.Keys.Contains(this.PrimaryMustContain.Replace("{{", string.Empty).Replace("}}", string.Empty)))
{
//assign the current value of the context param into our string
mustContain = e.WebTest.Context[this.PrimaryMustContain.Replace("{{", string.Empty).Replace("}}", string.Empty)].ToString();
}
else if (this.PrimaryMustContain.Contains("{{")) //this will only be evaluated if the context did not already contain the key
{
//throw to let them know we do not have their context param available to work with
throw new Exception("-- ERROR: ContextParameter Name='" + mustContain + "' was not found during processing in " + this.ToString());
}
//make sure context param exists
if (e.WebTest.Context.Keys.Contains(this.PrimaryMustNotContain.Replace("{{", string.Empty).Replace("}}", string.Empty)))
{
mustNotContain = e.WebTest.Context[this.PrimaryMustNotContain.Replace("{{", string.Empty).Replace("}}", string.Empty)].ToString();
}
else if (this.PrimaryMustNotContain.Contains("{{")) //this will only be evaluated if the context did not already contain the key
{
//throw to let them know we do not have their context param available to work with
throw new Exception("-- ERROR: ContextParameter Name='" + mustNotContain + "' was not found during processing in " + this.ToString());
}
}
//create our string array to contain the values we need to check for
string[] arrayPrimaryMustContain = mustContain.Split(new string[] { this.ListDelimiter }, StringSplitOptions.RemoveEmptyEntries);
string[] arrayPrimaryMustNotContain = mustNotContain.Split(new string[] { this.ListDelimiter }, StringSplitOptions.RemoveEmptyEntries);
//first we will check the request to see that it contains the required value
foreach (var item in arrayPrimaryMustContain)
{
if (e.Request.UrlWithQueryString.Contains(item.ToString(),this.IgnoreCase))
{
// the request is valid
e.Instruction = WebTestExecutionInstruction.Execute;
break; //exit for loop since we have a match, once valid it cannot be invalid
}
else
{
//the request does not contain the value we are searching for, exclude the request
e.Instruction = WebTestExecutionInstruction.Skip;
}
}
//now we will check the request for the case of not containing
//it should be noted that this check has precedence over the must contain in a conflict
foreach (var item in arrayPrimaryMustNotContain)
{
if (e.Request.UrlWithQueryString.Contains(item.ToString(),this.IgnoreCase))
{
// the request is valid
e.Instruction = WebTestExecutionInstruction.Skip;
break; //once invalid that is it
}
}
//if we skipped the instruction then add to comments
if (e.Instruction == WebTestExecutionInstruction.Skip)
{
if (!this.SuppressOutputComments)
{
e.WebTest.AddCommentToResult(this.ToString() + " forced skip of -> " + e.Request.UrlWithQueryString.ToString());
Debug.WriteLine(this.ToString() + " forced skip of -> " + e.Request.UrlWithQueryString.ToString());
}
}
Array.Clear(arrayPrimaryMustContain, arrayPrimaryMustContain.GetLowerBound(0), arrayPrimaryMustContain.Length);
arrayPrimaryMustContain = null;
mustContain = null;
}
}
public override void PostRequest(object sender, PostRequestEventArgs e)
{
base.PostRequest(sender, e);
if (!this.Enabled)
{
if (!this.SuppressOutputComments)
{
e.WebTest.AddCommentToResult(this.ToString() + " is currently disabled. No Processing was performed");
return;
}
}
//we only need to process if there are dependents
if (e.Request.DependentRequests.Count > 0)
{
//If RemoveAllDependentRequests is true we should never get here, since its wired up in hte PreRequest event
//to turn off parsing of dependents, so this is just a precaution
if (this.RemoveAllDependentRequests)
{
e.Request.DependentRequests.Clear();
//cleared now, bail out
if (!this.SuppressOutputComments)
{
e.WebTest.AddCommentToResult("Removed All Dependent Requests as specified by plug-in");
return;
}
}
//our private string vars to work with
string mustContain = this.DependentMustContain;
string mustNotContain = this.DependentMustNotContain;
//If they are using a context param, we need to fix this up for them.
//We know the context param format because it contains the "{{" as an example-> {{Parameter1}}.
if (mustContain.Contains("{{") || mustNotContain.Contains("{{")) //this is just a short circuit if we have no processing needed
{
//make sure the context param exists
if (e.WebTest.Context.Keys.Contains(this.DependentMustContain.Replace("{{", string.Empty).Replace("}}", string.Empty)))
{
//assign the current value of the context param into our string
mustContain = e.WebTest.Context[this.DependentMustContain.Replace("{{", string.Empty).Replace("}}", string.Empty)].ToString();
}
else if (this.DependentMustContain.Contains("{{")) //this will only be evaluated if the context did not already contain the key
{
//throw to let them know we do not have their context param available to work with
throw new Exception("-- ERROR: ContextParameter Name='" + mustContain + "' was not found during processing in " + this.ToString());
}
//make sure context param exists
if (e.WebTest.Context.Keys.Contains(this.DependentMustNotContain.Replace("{{", string.Empty).Replace("}}", string.Empty)))
{
mustNotContain = e.WebTest.Context[this.DependentMustNotContain.Replace("{{", string.Empty).Replace("}}", string.Empty)].ToString();
}
else if (this.DependentMustNotContain.Contains("{{")) //this will only be evaluated if the context did not already contain the key
{
//throw to let them know we do not have their context param available to work with
throw new Exception("-- ERROR: ContextParameter Name='" + mustNotContain + "' was not found during processing in " + this.ToString());
}
}
//create our string array to contain the values we need to check for
string[] arrayDependentMustContain = mustContain.Split(new string[] { this.ListDelimiter }, StringSplitOptions.RemoveEmptyEntries);
string[] arrayDependentMustNotContain = mustNotContain.Split(new string[] { this.ListDelimiter }, StringSplitOptions.RemoveEmptyEntries);
//list to hold the matches, later we will remove those
List<WebTestRequest> dependentsToRemove = new List<WebTestRequest>();
//we loop through each dependent request
foreach (var dependentRequest in e.Request.DependentRequests)
{
//assume the dependent request is not valid
bool IsDependentValid = true;
//we loop through our must contain array to check for a match
foreach (var itemMustContain in arrayDependentMustContain)
{
//if no match occurs we need to remove the dependent
if (dependentRequest.Url.Contains(itemMustContain,this.IgnoreCase))
{
//valid match - once valid, no more processing needs to occur in
//this loop
IsDependentValid = true;
break;
}
else
{
//not valid
IsDependentValid = false;
}
}
//we loop through our must not contain array to check for a match
//this rule will take precedence over the must contain
foreach (var itemMustNotContain in arrayDependentMustNotContain)
{
//if a match occurs we need to remove the dependent
if (dependentRequest.Url.Contains(itemMustNotContain,this.IgnoreCase))
{
IsDependentValid = false;
}
}
//add the dependent to the list if IsDependentValid false
if (!IsDependentValid)
{
dependentsToRemove.Add(dependentRequest);
}
}
//remove matches from the DependentRequests collection.
foreach (var item in dependentsToRemove)
{
e.Request.DependentRequests.Remove(item);
}
//if we removed some dependents, let them know we did
if ((dependentsToRemove.Count > 0) && (!this.SuppressOutputComments))
{
e.WebTest.AddCommentToResult("-- Removed " + dependentsToRemove.Count + " Dependent Requests as specified by " + this.ToString());
}
}
}
}
public static class ExtensionMethodsString
{
public static bool Contains(this string strValue, string strComparison, bool IgnoreCase)
{
if (IgnoreCase)
{
return strValue.ToLower().Contains(strComparison.ToLower());
}
else
{
return strValue.Contains(strComparison);
}
}
}
[Description("Utility plugin that allows you to skpip an individual request base on exsistence of a Context Paramater that you specify")]
public class SkipRequestContextParam : WebTestRequestPlugin
{
[Description("Allows you to supress comments output by this plugin by setting true/false")]
[DefaultValue(false)]
public bool SuppressOutputComments { get; set; }
[Description("Allows you to enable/disable this webtest plugin by setting true/false")]
[DefaultValue(true)]
public bool Enabled { get; set; }
[Description("Name of the context parameter that is required. If this Context Param does not exist this request will be skipped.")]
[DefaultValue("")]
public string ContextParameterName { get; set; }
[Description("If true, then the string length of the value contained in the Context Param Value must be at least 1 character. If there is no value set then the request will be skipped.")]
[DefaultValue(true)]
public bool RequireContextParamValue { get; set; }
public SkipRequestContextParam()
{
this.SuppressOutputComments = false;
this.Enabled = true;
this.ContextParameterName = string.Empty;
this.RequireContextParamValue = true;
}
public override void PreRequest(object sender, PreRequestEventArgs e)
{
base.PreRequest(sender, e);
if (!this.Enabled)
{
if (!this.SuppressOutputComments)
{
Debug.WriteLine(this.ToString() + " is currenlty disabled, no processing will occur.");
e.WebTest.AddCommentToResult(this.ToString() + " is currently disabled, no processing will occur.");
}
return; //no processing required
}
//if the request has already been set to skip, by another plugin for example, then there is no reason to
//execute the work in this plugin
if (e.Instruction == WebTestExecutionInstruction.Skip)
{
if (!this.SuppressOutputComments)
{
e.WebTest.AddCommentToResult(this.ToString() + " -> request instruction already set to Skip. No processing by this plugin occured.");
Debug.WriteLine(this.ToString() + " -> request instruction already set to Skip. No processing by this plugin occured.");
}
return;
}
//make sure the cotext param value has been set, otherwise thrown an exception
if (string.IsNullOrEmpty(this.ContextParameterName))
{
throw new ArgumentException("The Context Parameter Name Property was Null or Empty", "ContextParameterName");
}
//check for existence of the context param and skip if not found
if (!e.WebTest.Context.Keys.Contains(this.ContextParameterName))
{
e.Instruction = WebTestExecutionInstruction.Skip;
}
//check for length of string in the context param is not null or empty.
else if (string.IsNullOrEmpty(e.WebTest.Context[this.ContextParameterName].ToString()))
{
e.Instruction = WebTestExecutionInstruction.Skip;
}
//output the status of the plugin unless suppress output is true
if ((!this.SuppressOutputComments) && (e.Instruction == WebTestExecutionInstruction.Skip))
{
e.WebTest.AddCommentToResult(this.ToString() + " forced Skip of request->" + e.Request.Url);
Debug.WriteLine(this.ToString() + " forced Skip of request->" + e.Request.Url);
}
}
}
}