[INFO] Troubleshooting Pool Corruption

Hello All, time for another post, this time on Pool Corruption. Many of the STOP errors we in Support see are pool-related. For example, the one I saw some days ago:

X64_0x3B_nt!ExDeferredFreePool+249

Other common pool-related STOP errors include 0x1E, 0x50, 0xA, and 0x8E.

Quite often, the stack in such cases reveals an offending (third party) driver. Also often however, there is no clear indication of what might have happened. Take note also, that simply having a driver on the stack does not make it the culprit of the crash: it could have simply tripped over a corruption that was caused by another driver at an earlier point in time. In most cases, one memory dump is not enough, and we need additional information in subsequent dumps to find the root cause of the problem. We can obtain such info by enabling Driver Verifier. The options to specify related to Pool issues are Special Pool and Pool Tracking, under the Standard Flags:

STANDARD FLAGS:
These flags are considered standard options for Driver Verifier and can be
set using '/standard' or by the combination of the options: '/flags 0x209BB'
  bit 0 (0x00000001) - Special pool
  bit 1 (0x00000002) - Force IRQL checking
  bit 3 (0x00000008) - Pool tracking
  bit 4 (0x00000010) - I/O verification
  bit 5 (0x00000020) - Deadlock detection
  bit 7 (0x00000080) - DMA checking
  bit 8 (0x00000100) - Security checks
  bit 11 (0x00000800) - Miscellaneous checks
  bit 17 (0x00020000) - DDI compliance checking

The syntax to enable this: "verifier /flags 0x9 /all". This needs a reboot to be effective. Should you want to enable this run-time: "verifier /volatile /flags 0x9 /adddriver *". No reboot needed, but verification will also not be persistent across reboots. Please refer to the Driver Verifer link above for further information.

As a bonus, when enabling Pool Tracking, we also enable TrackNblOwner, which comes in handy when debugging network problems, as my friend and colleague Karim explains here.

Pool Tracking creates an in-memory database to store thread stacks related to pool allocations. By default, this is 65536 (0x10000) entries, but in cases where this is not enough, we can allocate up to 1048576 entries (0x100000), through the VerifierPoolTracesLength registry value under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

Note: only Windows 8 and higher. Each database entry is 128 bytes, so allocating one million (1048576) of these, only costs us 128MB of physical RAM, as the stack trace database is in Nonpaged Pool.

Thoughts? Comments? Let me know!