Troubleshooting Pool Leaks Part 6 – Driver Verifier
In part 5 we used poolhittag to get call stacks of pool being allocated and freed. This information is often essential to identifying the cause of a memory leak; however it is not always feasible to configure a live kernel debug to obtain this information. Fortunately there are alternative methods to get such call stacks.
Driver verifier has an option to enable pool tracking for a specific driver, or for multiple drivers. This functionality was first introduced in Windows Vista and Windows Server 2008. This information is also captured when driver verifier is used to enable special pool, however for the purposes of this article we will focus on using pool tracking.
The data stored by driver verifier requires a debugger to view. Any method of debugging can be used for this. You can use a live kernel debug as we described in part 4, you can get a memory dump (kernel or complete, a small dump is insufficient), or you can use livekd.
If you have used the steps from Part 1, Part 2, or Part 3, you likely have an idea which drivers are likely involved in creating the pool leak. In this example we are generating the leak using notmyfault, the same tool we have been using in prior examples. As seen in Part 2, the relevant driver is myfault.sys.
Although driver verifier has GUI, the easiest way to enable this functionality is with the below command from an elevated command prompt:
Verifier /flags 8 /driver myfault.sys
The above command will provide the following output, allowing you to confirm that the expected settings are enabled:
New verifier settings:
Special pool: Disabled
Pool tracking: Enabled
Force IRQL checking: Disabled
I/O verification: Disabled
Deadlock detection: Disabled
DMA checking: Disabled
Security checks: Disabled
Force pending I/O requests: Disabled
Low resources simulation: Disabled
IRP Logging: Disabled
Miscellaneous checks: Disabled
Verified drivers:
myfault.sys
You must restart this computer for the changes to take effect.
After rebooting the system, reproduce the memory leak and attach a debugger or generate a memory dump after the memory has been leaked.
Break in with the debugger (Ctrl+Break or Ctrl+C) or load the dump in windbg (File – Open Crash Dump).
Set the symbol path and reload symbols.
1: kd> .symfix c:\symbols
1: kd> .reload
Loading Kernel Symbols
...............................................................
The !verifier command has various options to view information about driver verifier. To view the pool allocations which have been tracked by verifier for notmyfault.sys, use the following:
0: kd> !verifier 3 myfault.sys
Verify Level 8 ... enabled options are:
All pool allocations checked on unload
Summary of All Verifier Statistics
RaiseIrqls 0x0
AcquireSpinLocks 0x0
Synch Executions 0x0
Trims 0x0
Pool Allocations Attempted 0xb
Pool Allocations Succeeded 0xb
Pool Allocations Succeeded SpecialPool 0xa
Pool Allocations With NO TAG 0x1
Pool Allocations Failed 0x0
Resource Allocations Failed Deliberately 0x0
Current paged pool allocations 0x0 for 00000000 bytes
Peak paged pool allocations 0x1 for 00000080 bytes
Current nonpaged pool allocations 0xa for 009CE000 bytes
Peak nonpaged pool allocations 0xa for 009CE000 bytes
Driver Verification List
Entry State NonPagedPool PagedPool Module
fffffa80031b5830 Loaded 009ce000 00000000 myfault.sys
Current Pool Allocations 0000000a 00000000
Current Pool Bytes 009ce000 00000000
Peak Pool Allocations 0000000a 00000001
Peak Pool Bytes 009ce000 00000080
PoolAddress SizeInBytes Tag CallersAddress
fffffa8005400000 0x000fb000 Leak fffff8800447d634
fffffa80052fb000 0x000fb000 Leak fffff8800447d634
fffffa8005200000 0x000fb000 Leak fffff8800447d634
fffffa80050fb000 0x000fb000 Leak fffff8800447d634
fffffa8005000000 0x000fb000 Leak fffff8800447d634
fffffa8004efb000 0x000fb000 Leak fffff8800447d634
fffffa8004e00000 0x000fb000 Leak fffff8800447d634
fffffa8004cfb000 0x000fb000 Leak fffff8800447d634
fffffa8004c00000 0x000fb000 Leak fffff8800447d634
fffffa8004a66000 0x000fb000 Leak fffff8800447d634
At the bottom of the above output is the list of allocations made by notmyfault.sys. For our purposes we are going to assume that these allocations have been leaked, as opposed to just being normal allocations that were not yet freed when the debugger was attached.
The !verifier command has an option to view call stacks for one of the tracked allocations. Keep in mind that the size of the database is limited and only more recent allocations will be kept in the database.
0: kd> !verifier 80 fffffa8005400000
Log of recent kernel pool Allocate and Free operations:
There are up to 0x10000 entries in the log.
Parsing 0x0000000000010000 log entries, searching for address 0xfffffa8005400000.
======================================================================
Pool block fffffa8005400000, Size 00000000000fa000, Thread fffffa80044ceb60
fffff80001927cc6 nt!VeAllocatePoolWithTagPriority+0x2b6
fffff80001927d3d nt!VerifierExAllocatePoolEx+0x1d
fffff8800447d634 myfault!MyfaultDeviceControl+0x358
fffff8800447d727 myfault!MyfaultDispatch+0xb7
fffff80001932750 nt!IovCallDriver+0xa0
fffff800017a7a97 nt!IopXxxControlFile+0x607
fffff800017a82f6 nt!NtDeviceIoControlFile+0x56
fffff8000148bed3 nt!KiSystemServiceCopyEnd+0x13
Parsed entry 0000000000010000/0000000000010000...
Finished parsing all pool tracking information.
The above output shows the call stack leading to the pool allocation. This is the same information we had seen in Part 5, however we are able to obtain this information using a dump or livekd, whereas the steps from Part 5 required an invasive debug and extended system downtime.
When you have completed troubleshooting, disable driver verifier with the following command and reboot:
verifier /reset