Share via

Troubleshooting Pool Leaks Part 2 – Poolmon

  • Article

In our previous article we discussed how to identify a pool leak using perfmon.  Although it may be interesting to know that you have a pool leak, most customers are interested in identifying the cause of the leak so that it can be corrected.  In this article we will begin the process of identifying what kernel mode driver is leaking pool, and possibly identify why.

 

Often when we are collecting data for a poor performance scenario there are two pieces of data that we collect.  Perfmon log data is one, as we discussed in our previous article.  The other piece of data is poolmon logs.  The memory manager tracks pool usage according to the tag associated with the pool allocations, using a technique called pool tagging.  Poolmon gathers this data and displays it in an easy to use format.  Poolmon can also be configured to dump data to a log, and in some scenarios it is beneficial to schedule poolmon to periodically collect such logs.  There are several available techniques to schedule poolmon, however that is beyond the scope of this article.

 

Poolmon has shipped with many different packages over the years; it is currently available with the Windows Driver Kit.  If you install the WDK to the default folder, poolmon will be in “C:\Program Files (x86)\Windows Kits\8.0\Tools\x64”.  Poolmon does not have dependencies on other modules in this folder; you can copy it to your other computers when you need to investigate pool usage.

 

How does pool tagging work?  When a driver allocates pool it calls the ExAllocatePoolWithTag API.  This API accepts a tag - a four-letter string - that will be used to label the allocation.  It is up to a driver developer to choose this tag.  Ideally each developer will choose a tag that is unique to their driver and use a different tag for each code path which calls ExAllocatePoolWithTag.  Because each tag should be unique to each driver, if we can identify the tag whose usage corresponds with the leak we can then begin to identify the driver which is leaking the memory.  The tag may also give the driver developer clues as to why the memory is being leaked, if they use a unique tag for each code path.

 

To view the pool usage associated with each tag run “poolmon -b” from a command prompt.  This will sort by the number of bytes associated with each tag.  If you are tracking pool usage over a period of time, you can log the data to a file with “poolmon -b -n poolmonlog1.txt”, replacing 1 with increasing numbers to obtain a series of logs.  Once you have a series of logs you may be able to view usage increasing for a specific tag, in a corresponding fashion to what you see in perfmon.

 

When analyzing poolmon the important data is at the top.  Typically the tag with the largest usage in bytes is the cause of the leak.

 

Poolmon -b

 

In the above data we can see that the tag with the most pool usage is “Leak”.  Now that we know what tag is leaking we need to identify what driver is using this tag.  Techniques for associating a leak with a tag vary, but findstr is often effective.  Most drivers are located in c:\windows\system32\drivers, so that is a good starting point when looking for the driver.  If you don’t find a result in that folder, go up a folder and try again, repeating until you get to the root of the drive.

 

C:\>findstr /s Leak *.sys

Users\Administrator\Desktop\myfault.sys:

AìI♦A╕Leak;┴☼B┴Aì

·∟   §£♂ Θ─☺  A╗☻   E☼"├Θ╡☺ Hï♣╔♂  ╞ $Θª☺  Hï♣:Hc┴ ┴ê\♦@ë

  └δ_Aï ╞♣@∟  ☺ë♣▓←  Aï@♦ë♣¼←  δCAâ∙♦u╓AïAìI♦A╕Leak;┴☼B┴3╔ï╨ §

 

In the above output we can see that “Leak” is used in myfault.sys.  If we hadn’t forced this leak with notmyfault, the next step in troubleshooting would be an internet search for the tag and the driver.  Often such a search will allow you to identify a specific fault within the driver and a solution.

 

Don’t panic if findstr doesn’t find your tag, or if you find the tag but it is not unique to one driver.  In future articles we will cover additional techniques for associating drivers with tags, and for associating allocations with specific code within a driver.

Comments

  • Anonymous
    September 02, 2012
    I prefer xperf for this. With stackwalking I can see more data compared to poolmon. [Xperf is very useful for leak analysis and we will have an article to cover this in detail.  Keep in mind that xperf is only useful if your leak happens quickly, as the .etl logs can grow large very quickly.  Also, because xperf logs typically only cover a short period of time it is useful to know what tag is leaking before starting xperf analysis.]

  • Anonymous
    December 26, 2012
    I have a big pool leak, but i can`t found the leakdriver... "MINI" ist the drivertag, can anyone help me? [Hi Lars.  We demonstrated various techniques to identify the driver using the tag in Part 4, 5, 6, and 7.  You can see the complete list of related articles at http://blogs.msdn.com/b/ntdebugging/archive/tags/pool+leak+series/]

  • Anonymous
    February 01, 2013
    @Lars MINI is used by the AMD GPU driver. Do you use Win8 and get the paged poll issue while playing BF3? Update to version 13.1 which includes the fix for this issue.

  • Anonymous
    April 26, 2013
    Hello Andre, I am facing simillar problems. On poolmon the file says Mini Nonp, is the the same driver whos fault it is to blame? I am currently running 13.4 on my 7970. [Hi Bluberr3. I am not Andre, but you can use the steps in our xperf article to determine what driver is allocating pool with the tag "Mini". http://blogs.msdn.com/b/ntdebugging/archive/2012/11/30/troubleshooting-pool-leaks-part-7-windows-performance-toolkit.aspx]

  • Anonymous
    August 08, 2013
    Hey, facing the same problems with like 5gb of ram in use of MINI nonp. got the 13.4 and got no clue, what causes the misuage. any help? [Hi Sonny.  This article is part of a series, please see Part 3 - 7 for different approaches to identifying the cause of the leak.  If you are using Windows 7/Server 2008 R2 or a newer OS, I recommend trying the steps in Part 7.]

  • Anonymous
    August 13, 2013
    The comment has been removed

  • Anonymous
    November 01, 2013
    Im having exactly the same issues as you guys describe - but only with Battlefield 4 in Multiplayer on Windows 8.1 Running poolmon.exe it's MINI which is causing a massive non paged memory usage (13GB out of 16 installed). Strange enough it won't flood the RAM directly but after couple of maps into the game. [The most effective way to find the source of your pool leak is probably to use xperf http://blogs.msdn.com/b/ntdebugging/archive/2012/11/30/troubleshooting-pool-leaks-part-7-windows-performance-toolkit.aspx]

  • Anonymous
    March 08, 2014
    I feel the same as matthias, I have 16 gb of ram and I was going all alone when I play battlefield 4 with windows 8.1, when running poolmon MINI nonp me jump. I need help, thanks [Xperf can be used to identify what driver is allocating this pool.  For more information see http://blogs.msdn.com/b/ntdebugging/archive/2012/11/30/troubleshooting-pool-leaks-part-7-windows-performance-toolkit.aspx and http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-48-WPT-Memory-Analysis-Pool]

  • Anonymous
    March 08, 2014
    I feel the same as matthias, I have 16 gb of ram and I was going all alone when I play battlefield 4 with windows 8.1, when running poolmon MINI nonp me jump. I need help, thanks

  • Anonymous
    November 21, 2015
    Hi All, We have Windows server 2008, where kernel memory usage keeps on growing; Looking at the poolmon output could see "FSEL" tag is leaking the memory. Any idea what this tag is all about and how to fix this issue; [I don't know what FSEL is.  FSel (different capitalization) is used by the FsRtlAllocateExtraCreateParameterList API, https://msdn.microsoft.com/en-us/library/windows/hardware/ff545632.aspx .]