Hotfix to Enable Mini-Filter Performance Diagnostics With XPerf for Windows Server 2008R2
Greetings ntdebugging community, Bob here again and today I would like to let everyone know about a new feature implemented in Windows Server 2008 R2’s kernel and filter manager binaries released in knowledge base article 2666390.
Beginning with this update, a minifilter that is adversely affecting system performance can be identified in Windows 2008 R2. These measurements are taken within the filter manager and measure the execution time of the minifilter’s pre and post operations. On every file access file system minifilters are given the chance to perform pre and post modifications to IO operations within the driver stack.
To start recording the operations and to view the results, the latest version of the Windows Performance Toolkit from the Windows 8 beta ADK needs to be installed on the system. How to install the package can be found in the KB article.
The command to start recording and to record stack traces is (note that these commands may wrap depending on your screen resolution, they are intended to be typed in one line):
xperf -on PROC_THREAD+LOADER+FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE+FILENAME -stackwalk MiniFilterPreOpInit+MiniFilterPostOpInit
The command above starts a trace with most of the buffering done in memory. The command below specifies a file used as a backing store to use less memory:
xperf -start -f <traceFile> -on PROC_THREAD+LOADER+FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE+FILENAME -stackwalk MiniFilterPreOpInit+MiniFilterPostOpInit
Note: <traceFile> is the path of the file to store the backing file (i.e. trace.etl). It is recommended that the backing file be placed on a drive other than the one experiencing the problem, to avoid xperf operations interfering with or contributing to the scenario you are measuring. You can add the +FILENAME switch if you want to log names of the files that the minifilter is operating on.
You may not know if a minifilter is causing the problem until after you begin data analysis, so you can add the filter manager tracing to the Driver Delays command presented in my previous blog, Using Xperf to investigate slow I/O issues.
xperf -on PROC_THREAD+LOADER+CSWITCH+FILENAME+FILE_IO+FILE_IO_INIT+DRIVERS+FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE -f kernel.etl -stackwalk CSwitch+DiskReadInit+DiskWriteInit+DiskFlushInit+FileCreate+FileCleanup+FileClose+FileRead+FileWrite+MiniFilterPreOpInit+MiniFilterPostOpInit -BufferSize 1024 -MaxBuffers 1024 -MaxFile 1024 -FileMode Circular
To stop the trace and flush data to the trace file, type the following command:
xperf –stop –d mymergedtrace.etl
The file specified must be a different file name than the one provided with the –f switch if that option was used when starting the trace.
Before we begin looking at the trace, let’s talk a little about what we are tracing. We are tracing fltmgr calling into the minifilter. The fltmgr manages minifilters. When a file operation is done, each minifilter has a chance to process and optionally modify an operation on a file before and after the call goes into the file system. For example, in this case we are going to see results of the time it took for the minifilter to do a “pre create” operation. That is the function that is called before the “create file” function is sent to the file system. This tracing can also track the time it took for the minifilter to do a “post create” operation. That is the minifilter function that gets called after the create file is sent to the file system. So the minifilter is called before and after every file system operation. The minifilter may not have functions to process before and after, however it has the option.
Once the problem is traced, the data can be viewed by clicking on the merged etl file and you will see a similar presentation as below.
The Mini-Filter Delays are in the Storage display. So click on the triangle on the left side of Storage and you will see:
To get more detail, right click on the “Mini-Filter Delays” section and select “Add To New Analysis View”and that will bring up minifilter delays detail as below.
The Duration is the time in micro seconds of the total time delay in the minifilter. So in the example above the scanner minifilter did 4,068 requests and it took 30 seconds total.
We can expand the display by clicking on the triangle next to scanner.sys to get a more detailed view, I chose to investigate scanner.sys because it has the longest Duration. I dragged the “Major Function Name (Count)” and “File Name” columns to the left of the yellow bar to get detail and used the triangles to expand the ProcessID with the longest Duration, and then I expanded the File Name with the longest Duration. To get more usable space on your monitor, you can click the X to close the Graph Explorer (to get it back, from the Window menu choose Graph Explorer).
On the display above it shows the scanner.sys filter was operating on the file “C:\users\Administrator\Desktop\scanner\test2.txt” The green 6 is the total operations. So there were three cleanups and three creates. So the display is telling us that the filter did 3 “create” operations that took close to 15 seconds and 3 cleanup operations that took 5 milliseconds on the file.
If you prefer operating from a command line rather than a GUI, this same data can be extracted using the below command. Note that -min specifies the time in ns, so this command will show delays longer than 100 ms.
xperf -i mymergedtrace.etl -a minifilterdelay -min 100000
This will give you output similar to the below information:
[1/2] 100.0%
[2/2] 100.0%
There are 3 MiniFilter Delay completion events in given time range.
Process Name, Module Name, Call (us), Return (us), Time (us), Major Func Name, Filename
explorer.exe, scanner.sys, 18649145, 23645502, 4996357, CREATE, C:\Users\Administrator\Desktop\scanner\test2.txt
explorer.exe, scanner.sys, 23646355, 28644638, 4998282, CREATE, C:\Users\Administrator\Desktop\scanner\test2.txt
explorer.exe, scanner.sys, 28645194, 33645158, 4999964, CREATE, C:\Users\Administrator\Desktop\scanner\test2.txt
It seems there is a problem in the create file handling of the scanner.sys minifilter. For some reason the create file function in the minifiltertook 5 seconds for each operation. The vendor of this filter would need to do additional investigation to identify why the filter has this delay.
Comments
Anonymous
June 28, 2012
Do you plan to make this hotfix available for Windows 7? [The fix works on Windows 7. I'll update the article to reflect this.]Anonymous
June 09, 2014
The comment has been removed