Off Road Profiling of ASP.NET Applications

Article
05/10/2005

David Gray, Developer, Dynamic Analysis Team, VSTS

Introduction:

The VSTS profiler team sweated blood for beta2 to get the 'click and go' profiling of ASP.NET application to work smoothly. I believe the effort paid off and for mainline developer scenarios, you should be able to open a website in visual studio, run the peformance wizard and click the launch button. (Okay, maybe that's more than one click, but still..) If you want to try this, first take a look at Eric Jarvi's article ASP.NET Profiling

But there will always be the happy developer out there (especially using a beta product) that has a configuration we didn't consider 'mainline' or has a need to gather data in a way that isn't affected by all the things we have to do to make the user experience work for the 'click and go' solution. One obvious example is having Visual Studio running on the machine while gathering performance data has some effect. Another is that we force a shutdown of IIS right before we start gathering data. In this article I'm going to show you how to go off the beaten track a bit and gather performance data with our tools by hand (so to speak). For now, I'm going to assume you have a full install of VSTS on the machine that you're running the profiler on. But don't worry, you won't have to actually run Visual Studio while you're gathering data.

What Profiling Mode Do I Need? Sampling or Instrumentation?

This is just about always the first question to ask when setting up to gather data using our tools. So there quite a bit of blogging going on about the differences between sampling and instrumented profiling. See https://blogs.msdn.com/profiler for our team blog and related links. If you're interested in understanding the difference in depth, I highly recommend spending the time to go through our team blogs

But for now, here are the CliffsNotes®. Normally, I'd tell you to use sampling profiling to get a quick overview of how your app is behaving and hopefully narrow your problem down to hot-spots. Then use instrumentation profiling to take a closer look at the assembly(s) that contain the hot-spots identified by sampling.

But wait - you're running an ASP.NET application. When sampling you end up with a whole bunch of information about what’s going on in the ASP.NET framework that will most likely obscure what is going on in your own code. So for ASP.NET profiling, I recommend reversing the process. Try doing instrumented profiling on your own code and only if you can’t find anything there, move to sampling profiling to see if interaction with the ASP.NET framework might be your performance culprit.

What exactly do I want to profile?

ASP.NET is a complex system. Your code is only a small part of what is going on even if you are running a pretty big web application. For our purposes, there are two general categories of user code in ASP.NET applications.

Pre-built Assemblies - generally placed in the bin directory of the web site
Dynamically Built Assemblies - for example from code embedded in web pages and within the App_Code directory.

The following sections are perscriptive guidance on how to gather data for these types of data in each profiling mode. The careful reader will note that there are only three sections rather than the expected four. This is because in sampling mode, we gather data on the entire process, so the type of assembly you're interested in isn't relevant.

All of these sections assume that Visual Studio Team Systems is installed on your machine.

So put on you sturdy hiking boots, make sure you're not wearing anything that you mind getting muddy, and join me off the beaten track in one of these 'off-road' experiences.

--dwg

Instrumented Profiling of Pre-built Assemblies:

These are assemblies that are placed in the bin directory of the web site and so are not dynamically built by ASP.NET.

vsperfclrenv /globaltraceon
- This will set up the environment variables that tell .NET to load our profiling helpers.
reboot (as of this writing reboot is still required)
vsinstr [my assembly path]
- vsinstr should be run on any pre-built assemblies that you wish to gather performance data on.
vsperfmon /TRACE /OUTPUT:[your ouputfile].vsp /USER:"[ASP.NET worker process user]"
- This will start the performance monitor that will collect the data as your application runs.
- See Appendix A to find the user name that the ASP.NET worker process uses (depends on version of IIS you have running).
- vsperfmon will run throughout the collection process, so you must start another command prompt to complete other commands.
Run your test scenario against the web application
iisreset /STOP
- The profiled process must be shut down for collection to complete
vsperfcmd -SHUTDOWN
vsperfreport [your outputfile].vsp /SUMMARY:FUNCTION /PACKSYMBOLS
- PACKSYMBOLS is necessary to get the symbols pinned correctly before the target assemblies are lost from their location in temporary storage, it also preserves symbols so the report can be meaningful if opened on another machine for analysis.
iisreset /START
- Only do this if you are going to do more profiling
vsperfclrenv /globaloff
- This step and the following step will prevent our profiling helpers from being loaded into ASP.NET in the future. This should definitely be done when you have completed profiling, as loading our helpers can affect performance.
reboot
Open [your outputfile].vsp in Visual Studio.
- Since you packed the symbols in a previous step, you can copy [your outputfile].vsp to any machine that has a full install of VSTS and open in there.

Instrumented Profiling of Dynamically Built Assemblies

These are all of the assemblies that contain code from your web pages and code behind and App_Code directories.

vsperfclrenv /globaltraceon
- This will set up the environment variables that tell .NET to load our profiling helpers.
reboot
Back up the web.config file
Fix up the web.config file for your site
- See Appendix B for instructions on fixing up web.config
vsperfmon /TRACE /OUTPUT:[your ouputfile].vsp /USER:"[ASP.NET worker process user]"
- This will start the performance monitor that will collect the data as your application runs.
- See Appendix A to find the user name that the ASP.NET worker process uses.
- vsperfmon will run throughout the collection process, so you must start another command prompt to complete other commands.
Run your test scenario against the web application.
iisreset /STOP
- The profiled process must be shut down for collection to complete
vsperfcmd -SHUTDOWN
vsperfreport [your outputfile].vsp /SUMMARY:FUNCTION /PACKSYMBOLS
- PACKSYMBOLS is necessary to get the symbols pinned correctly before the target assemblies are lost from their location in temporary storage, it also preserves symbols so the report can be meaningful if opened on another machine for analysis.
Restore the web.config file (if you don't intend to do more profiling)
iisreset /START
vsperfclrenv /globaloff (if you don't intend to do more profiling)
- This step and the following step will prevent our profiling helpers from being loaded into ASP.NET in the future. This should definitely be done when you have completed profiling, as loading our helpers degraded performance.
reboot
Open [your outputfile].vsp in Visual Studio.
- Since you packed the symbols in a previous step, you can copy [your outputfile].vsp to any machine that has a full install of VSTS and open in there.

You can profile both pre-built and dynamically generated .assemblies at the same time by following the instructions for dynamically built assemblies but manually running vsinstr against the prebuilts prior to fixing the web.config file.

Sampling Profiling

vsperfclrenv /globalsampleon
- This will set up the environment variables that tell .NET to load our profiling helpers.
reboot
vsperfmon /SAMPLE /OUTPUT:[your ouputfile].vsp /USER:"[ASP.NET worker process user]"
- This will start the performance monitor that will collect the data as your application runs. See Appendix A to find the user name that the ASP.NET worker process uses. Vsperfmon will run throughout the collection process, so you must start another command prompt to complete other commands.
vsperfcmd /ATTACH:[PID of ASP.NET WORKER PROCESS]
- See Appenendix A to find the process id of the ASP.NET worker process.
Run your test scenario against the web application.
- It is important to run a scenario that stresses the CPU in order to get usable results out of sampling profiling. Try to get the CPU to a 100% for several minutes. Using the new Whidbey load test tools can be good for this.
vsperfcmd /DETACH
- This stops collection of data
vsperfcmd /SHUTDOWN
- This tells the performance monitor that you are done with your scenario and to close up the log file.
vsperfreport [your outputfile].vsp /SUMMARY:FUNCTION /PACKSYMBOLS
- This packs the symbols into your output file so that they can be transported to another machine for analysis. This is also necessary to get the symbols pinned correctly before the target assemblies are rebuilt.
vsperfclrenv /globaloff
- This step and the following step will prevent our profiling helpers from being loaded into ASP.NET in the future. This should definitely be done when you have completed profiling, as loading our helpers degraded performance.
reboot
Open [your outputfile].vsp in Visual Studio.
- Since you packed the symbols in a previous step, you can copy [your outputfile].vsp to any machine that has a full install of VSTS and open in there.

Appendix A: Finding the User Name and Process Id for the ASP.NET Worker Process

Make certain that your ASP.NET application is running
Open the task manager
Go to the Processes Tab
Show the PID and User Name columns
- Do this by going to the view menu and choosing select columns and select PID and User Name
If aspnet_wp.exe exists use it, otherwise look for w3wp.exe.

  On win2K3Srv there may be multiple worker processes depending on how isolation is set up.  In this case, use %windir% \system32\iisapp.vbs to correlate w3wp PIDs with IIS application pools.

Read the PID and User Name Columns (defaults listed below)
- Windows Server 2003/IIS 6: (w3wp.exe, NETWORK SERVICE)
- Windows XP/IIS 5.1: (aspnet_wp.exe, ASPNET)
- Windows 2000/IIS 5.0: (aspnet_wp.exe, LOCAL SYSTEM)

Appendix B: Modifying the web.config File

1. Add or modify the compilation tag:

  <system.web>
 <compilation assemblyPostProcessorType="Microsoft.VisualStudio.Enterprise.Common.AspPerformanceInstrumenter,  Microsoft.VisualStudio.Enterprise.ASPNetHelper, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
 </system.web>

NoteThe PublicKeyToken must match the PublicKeyToken of the ASPNetHelper.dll.

2. Set up the runtime tag to show the location of <runtime> <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1"> <dependentAssembly> <assemblyIdentity name="Microsoft.VisualStudio.Enterprise.ASPNetHelper" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" /> <codeBase version="8.0.0.0" href="file:///C:/Program%20Files/Microsoft%20Visual%20Studio%208/Common7/IDE/PrivateAssemblies/Microsoft.VisualStudio.Enterprise.ASPNetHelper.DLL" /> </dependentAssembly> </assemblyBinding> </runtime>

NoteThe PublicKeyToken must match the PublicKeyToken of the ASPNetHelper.dll. Also, the HREF in codebase must be a file URL (not just a pathname) pointing to this .DLL.

3. Set up the location of vsinstr and its dependencies

  <appSettings>
 <add key="Microsoft.VisualStudio.Enterprise.AspNetHelper.VsInstrLocation" value="C:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools\vsinstr.exe" />
 <add key="Microsoft.VisualStudio.Enterprise.AspNetHelper.VsInstrTools" value="C:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools\" />
 </appSettings>

VsInstrLocation is the full path including fil ename of vsinstr.exe.
VsInstrTools is the directory containing helper .dlls for vsinstr that aren’t available elsewhere on the path, these are msdis150.dll and mspdb80.dll.

Here is a complete sample web.config

  <?xml version="1.0"?>
 <configuration>
 <runtime>
 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
 <dependentAssembly>
 <assemblyIdentity name="Microsoft.VisualStudio.Enterprise.ASPNetHelper" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
 <codeBase version="8.0.0.0" href="file:///C:/Program%20Files/Microsoft%20Visual%20Studio%208/Common7/IDE/PrivateAssemblies/Microsoft.VisualStudio.Enterprise.ASPNetHelper.DLL" />
 </dependentAssembly>
 </assemblyBinding>
 </runtime>
 <system.web>
 <compilation assemblyPostProcessorType="Microsoft.VisualStudio.Enterprise.Common.AspPerformanceInstrumenter, Microsoft.VisualStudio.Enterprise.ASPNetHelper, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
 </system.web>
 <appSettings>
 <add key="Microsoft.VisualStudio.Enterprise.AspNetHelper.VsInstrLocation" value="C:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools\vsinstr.exe" />
 <add key="Microsoft.VisualStudio.Enterprise.AspNetHelper.VsInstrTools" value="C:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools\" />
 </appSettings>
 </configuration>

Appendix C: Troubleshooting

SymptomEverything seemed to work, but when I tried to open the file I got an error that ‘File contains no data’

There are two likely candidates for this.

The /USER switch was not correctly set when starting vsperfmon.exe. Check and make certain that you correctly identified the process identity of the ASP.NET worker process (see appendix B) and make certain that you provide it to vsperfmon.exe.
No assemblies were instrumented or the instrumented assemblies were never run. The command prompt that hosts vsperfmon.exe will show a message when it starts getting data from an instrumented process. If you never see that message, something went wrong. Make certain that the web.config file was correctly modified. If you manually instrument an assembly, make certain it is the one that is being run.

SymptomMy web page shows an error that vsperf80.dll can’t be found

This is because vsperf80.dll must be on the path for the ASP.NET worker process. Both the Visual Studio setup and the stand-alone performance tools setup should place vsperf80.dll in [Windows]\system32.

locate vsperf80.dll and place in your [Windows]\system32 directory.

SymptomEverything seemed to work, but I'm not seeing all the symbols I expect.