ASP.NET Internals - Performance counters implementation details
Introduction
Many users seem to run into issues when using ASP.NET Performance counters. With that in mind, I decided to write this article to explain, in as much detail as possible, how the ASP.NET Performance counters were implemented. In the hope people will find this useful when dealing with such problems. This might be also useful for support engineers helping customers.
The basis.
The first thing to know is how counters were implemented in the Os at the time of ASP.NET performance counters initial coding. This is explained in great detail in the msdn web site. Don’t worry. You don’t need to read all that documentation to understand the basis of how this works.
Basically, what you need, in order to provide performance counters, is a native dll with at least three exports: Open, Close and Collect. You also need an .ini file with the counters names and descriptions, and an .h file with the indexes.
You also need a registry key under HKLM\SYSTEM\CurrentControlSet\services\<your_app>\Performance. This key should contain the “Library”, “Open”, “Close” and “Collect” values with your native dll and its exports.
Once this is ready, you can call lodctr.exe (or LoadPerfCounterTextStrings() from your code) passing the .ini file path as a parameter. This will register the performance counters. The call will populate some other registry values on the registry key mentioned before. The performance counters names are stored at HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib\nnn, where nnn is the language id. You can use unlodctr.exe (or UnloadPerfCounterTextStrings()) to unregister. Errors during this process are usually stored in the event log.
Versioned versus Generic
ASP.NET Performance counters has generic categories (e.g. ASP.NET and ASP.NET Applications) and versioned categories (e.g. ASP.NET v2.50727 and ASP.NET Apps v2.50727). This allows side-by-side versions with different performance counters implementation. The higher version registered on the machine “owns” the generic objects, while all versions “owns” their respective versioned objects.
Note that if you have an ASP.NET application running in a lower version, then the generic counters will not report numbers for that application. You need to use the versioned categories in that case. For instance, if you have both 2.0 and 4.0 registered in the same machine, and a web application is running in an application pool using 2.0, then the ASP.NET and ASP.NET Application categories will not report data for that app. You need to use ASP.NET v2.50727 and ASP.NET Apps v2.50727 categories instead.
Global versus Instance-based.
ASP.NET has some counters that are global. That means that they are aggregated for all web apps running in that version of ASP.NET. These counters are reported thru the ASP.NET category (or ASP.NET vx.xxxxx).
Instance based counters can be obtained for a particular app. These counters are accessible with the ASP.NET Applications category (or ASP.NET Apps vx.xxxxx, for versioned). The instance name is the app domain id of the app. When hosted in IIS, this ID is made with the “metabase” path (e.g. _LM_W3SVC_1_ROOT_MYAPP). When running in Visual Studio, the ID is a hex integer, created with a hash code of a combination of virtual and physical path of the app.
ASP.NET performance counters files.
In the case of ASP.NET 2.0, the native library providing performance counters is called aspnet_perf.dll and it resides in the framework directory (e.g. C:\Windows\Microsoft.NET\Framework64\v2.0.50727). This library implements the exports for the generic performance counters (e.g. ASP.NET), the versioned performance counters (e.g. ASP.NET_2.0.50727) and the aspnet_state service performance counters. Older versions had the export methods directly on the aspnet_isapi.dll.
In 4.0 we added a new library called aspnet_counters.dll. This library resides in the System32 directory (e.g. c:\Windows\System32 and c:\Windows\SysWOW64). It replaced aspnet_perf.dll in the registration, but it’s actually a shim, which redirects all calls to aspnet_perf.dll. The later still is the one doing all the work.
The problem with the previous registration design was that a full path was required in the “Library” value in both the versioned and the generic keys. This created several problems. First, you needed separated registry keys for 64 bit and 32 bit. Also, in Windows 2003, the performance counters will sometimes stop working when using Windows Management Instrumentation (WMI). Additionally, 32 bit clients (e.g. mmc.exe /32 perfmon.msc) couldn’t access performance counters for the session state service.
Other files used by performance counters in the framework directory are the followings: The aspnet_perf2.ini file is used for generic counters, while aspnet_perf.ini is used to register the versioned counters. They share the same aspnet_perf.h. The state service files are aspnet_state_perf.ini and aspnet_state_perf.h.
ASP.NET performance counters registry keys.
ASP.NET has multiple registry keys under HKLM\SYSTEM\CurrentControlSet\services. The list below shows all the keys and their use:
· ASP.NET: Registry key for the generic counters. When 2.0 is the highest version registered, this will contain the registration for 32 bit clients. In 4.0 this key is always platform agnostic, so both 64 and 32 clients can use this entry.
· ASP.NET_64: Registry key for the generic counters on 64 bit Os. When 2.0 is the highest version registered, this will contain the registration for 64 bit clients. In 4.0 and above, this key will be deleted by the registration process.
· ASP.NET_2.0.50727: Registry key for the 2.0 versioned counters. It contains the registration for 32 bit clients.
· ASP.NET_64_2.0.50727: Registry key for the 2.0 versioned counters. It contains the registration for 64 bit clients.
· ASP.NET_4.0.30319: Registry key for the 4.0 versioned counters. The key is platform agnostic. It will work with any client, regardless of its bitness.
· aspnet_state: Contains registration for session state performance counters. In 64 bit Os, only 64 bit library will be present.
Each key also contains sub-keys. Their use is described below:
a) Performance: Sub-key with “Library” and exports. Other values are set by the Os.
b) Names: Pipe names. One per each running worker process. See “What happens at runtime?”. This sub-key appears only on the 32 bit versioned keys.
c) Linkage: In 4.0 and above we store here the referenced version number of the ASP.NET registered, in a value called “Export”. This sub-key was not present or used in ASP.NET 2.0.
d) Parameters: Not related to performance counters. Only on the session state service.
The registration process.
The registration of the performance counters is executed from within aspnet_regiis.exe tool. Registration happens with switches -i, -iru, -ir or -update (“undocumented”). During framework setup this tool is actually called as a custom action by the installation framework. The parameter used during setup is -iru. Note that starting with Vista, since the .Net framework is part of the Os, the performance counters can also be registered during package installation (e.g. pkgmgr /iu:IIS-WebServerRole;…;IIS-ASPNET). This is done by a tool called aspnetca.exe, in the IIS folder (e.g. %windir%\System32\inetsrv), which does a job similar to that of aspnet_regiis.exe.
To register the generic counters, aspnet_regiis.exe first calls UnloadPerfCounterTextStrings() with “ASP.NET” as parameter. This ensures we always have a clean registration. Additionally, in 4.0 and newer, we also unregister the “ASP.NET_64” driver and remove the associated registry key. Then aspnet_regiis.exe calls LoadPerfCounterTextStrings() passing the file path to aspnet_perf2.ini as parameter.
For versioned counters, we also unregister first (via UnloadPerfCounterTextStrings()) and then register with aspnet_perf.ini as parameter. This is done every time you call aspnet_regiis.exe with any registration switch.
The state service counters are a little bit different. They are only registered together with the service, so once this is registered, the performance counters are not unreg/reg again in subsequent calls to aspnet_regiis.exe. Also, under a 64 bit Os, the counters are not registered if you run the 32 bit version (wow64 process) of aspnet_regiis.exe.
When unregistering, aspnet_regiis.exe has to make sure the next available registered version of ASP.NET takes ownership of the generic counters. So after we call UnloadPerfCounterTextStrings() for the generic counters, we search the registry trying to find another version of ASP.NET in the HKLM\SOFTWARE\Microsoft\ASP.NET registry. If one is found, we call LoadPerfCounterTextStrings() with that version’s aspnet_perf2.ini. For instance, if both 2.0 and 4.0 are registered, then, calling aspnet_regiis.exe -u from the 4.0, would actually associate the 2.0 version with the generic counters. Versioned counters will simply be unregistered for that version.
The aspnet_regiis.exe uses log text files to store traces of the operations performed. The logs are stored in the %TEMP% folder and are named something like ASPNETSetup_00xxx.log. These files can be examined, along with the event log when looking for clues of what went wrong.
Running aspnet_regiis.exe -iru from the latest version is the safest and recommended workaround when trying to fix issues with ASP.NET performance counters.
What happens during runtime?
Internally, ASP.NET performance counters mechanism uses a client/server approach. In this case, the server is the worker process; say w3wp.exe, while the clients are other processes that consume performance counter data using Win32 API, such as PDH, registry functions (e.g. perfmon), or a higher level API such as System.Diagnostics or WMI.
During runtime, whenever a web application is run and the worker process starts, it loads webengine.dll (or webengine4.dll in .Net 4.0). This library, among other many things, will initialize the performance counters local storage in the process. It will also create a named pipe to which clients will be able to connect and read the counters data from. The name of the pipe is stored in the HKLM\SYSTEM\CurrentControlSet\services\ASP.NET_x.x.xxxxx\Names registry key, so clients can find it.
The processing of the named pipe is asynchronous. The library uses an IO completion thread from the CLR, which wakes up every time some client wants to get performance counter data. Other than that, the feature doesn’t consume much cycles, except when some part of ASP.NET runtime wants to set or increment a performance counter value. For instance, when a request comes, we need to increase the value of the "ASP.NET\Requests Current" counter. When the request is done, we decrement it.
When a client such as perfmon, wants to get performance counters data, using their API of choice, the Os will loop through all registered counters at HKLM\SYSTEM\CurrentControlSet\services, calling the Open export. This will eventually call the aspnet_perf.dll, which will initialize the client side of the connection. Among other things, the library will create two background threads.
One thread monitors the HKLM\SYSTEM\CurrentControlSet\services\ASP.NET_x.x.xxxxx\Names registry key, so it will be informed of any new worker process.
The second thread will enter a loop, waking up every 400 milliseconds (or sooner, if instructed by a call to the Collect export) and reading data thru the named pipes from all worker processes. The data is stored in a global area, and it will be returned to the clients through the Collect export. That call usually executes on the main thread of the client process. A locking mechanism is used to protect the shared memory.
If an error is detected during the reading of the pipes, the error is ignored and the data is cleared, just in case the worker process dies unexpectedly. So, counters could report zeroes when running on heavy load.
When transferring data thru the pipe, the client will first send its version number to the worker process, which will validate that it matches its own. If it does it will send the performance counter data back.
Any error during clients calls to Collect should be stored in the event log. There is not internal reporting for errors on the gathering threads.
Conclusion.
Hopefully, the information contained here will help somebody. As usual, I can’t really see what the possible uses of the information presented here are. In any case, I hope it will benefit someone.
Additional Resources
Other performance related articles: https://blogs.msdn.com/josere
Performance counters in msdn: https://msdn.microsoft.com/en-us/library/aa373083(VS.85).aspx
OpenPerformanceData: https://msdn.microsoft.com/en-us/library/aa372200(VS.85).aspx
CollectPerformanceData: https://msdn.microsoft.com/en-us/library/aa371898(VS.85).aspx
ClosePerformanceData: https://msdn.microsoft.com/en-us/library/aa371895(VS.85).aspx
lodctr: https://msdn.microsoft.com/en-us/library/ms894317.aspx
unlodctr: https://msdn.microsoft.com/en-us/library/ms929082.aspx
LoadPerfCounterTextStrings: https://msdn.microsoft.com/en-us/library/aa372187.aspx
UnloadPerfCounterTextStrings: https://msdn.microsoft.com/en-us/library/aa373205(VS.85).aspx
Comments
Anonymous
June 22, 2010
Nice Article. And also clearly explained.Anonymous
June 22, 2010
Great Article. Thanks for share with usAnonymous
July 04, 2011
Nice Article. It surely helped me. Thanks.Anonymous
November 27, 2012
Nice Article.Anonymous
December 12, 2012
very nicely explained, and very very useful. Thank you for publishing.Anonymous
August 14, 2013
Thanks for the article. I wonder if you'd be able to shed any light on an issue with the Cache Total Entries value in Windows Server 2008 R2 - it seems to show a throughput value rather than a total value (works fine on Windows 8). If you are interested, check out: forums.asp.net/.../1Anonymous
November 29, 2013
nice article....