Identifying Global Atom Table Leaks

Hi, it's the Debug Ninja back again with another debugging adventure.  Recently I have encountered several instances where processes fail to initialize, and a review of available resources showed that there was no obvious resource exhaustion.  A more in depth review found that there were no available string atoms in the global atom table.

 

Global atoms are organized on a per-session basis.  If atoms cannot be allocated in session 0, services may fail to start or processes launched by various services may fail to start.  However, a user logged in to a different session will not experience any such failures.

 

String atoms are numbered from 0xC000 through 0xFFFF, providing a maximum of 0x4000 atoms per session.  For more information on atoms, and atom tables, see https://technet.microsoft.com/en-us/query/ms649053.

 

When there are no more string atoms available, calls to APIs that allocate string atoms will fail.  Because atoms are often allocated at process or dll init time, the most common symptom is that processes fail to initialize.  The process may cleanly exit without an error.  You are likely experiencing this problem if you debug your application and find that the failure originates from an API that allocates string atoms such as RegisterClass, RegisterClassEx, GlobalAddAtom, or AddAtom.

 

To determine if the global string atom table is full you will need to perform a kernel debug.  This can be a live debug or a post-mortem debug using a dump.

 

First identify the session where the failures have occurred and set the process context to a process in this session.  In my example, w3wp.exe was launching a process and this process failed to initialize.

 

2: kd> !process 0 0 w3wp.exe

PROCESS fffffa8005083060

    SessionId: 0  Cid: 1668    Peb: fffdf000  ParentCid: 08ec

    DirBase: 8a2df000  ObjectTable: fffff8a0128bbe40  HandleCount: 441.

    Image: w3wp.exe

2: kd> .process /p /r fffffa8005083060

Implicit process is now fffffa80`05083060

Loading User Symbols

.....

 

Next we need to analyze the global atom table.  The pointer to the table is stored in the UserAtomTableHandle global.

 

2: kd> dq win32k!UserAtomTableHandle l1

fffff960`003bf7a8  fffff8a0`05e5bc70

 

The UserAtomTableHandle has a pointer to a handle table at offset 0x10 in 64-bit, and offset 0x8 in 32-bit.  Note that although the atom table is defined as a _RTL_ATOM_TABLE, the format shown by dt is for user mode and does not apply to the UserAtomTableHandle in kernel mode.

 

2: kd> dq fffff8a0`05e5bc70+10 l1

fffff8a0`05e5bc80  fffff8a0`05db7740

2: kd> dt nt!_HANDLE_TABLE fffff8a0`05db7740

   +0x000 TableCode        : 0xfffff8a0`109c8001

   +0x008 QuotaProcess     : (null)

   +0x010 UniqueProcessId  : 0x00000000`00000184 Void

   +0x018 HandleLock       : _EX_PUSH_LOCK

   +0x020 HandleTableList  : _LIST_ENTRY [ 0xfffff8a0`05db7760 - 0xfffff8a0`05db7760 ]

   +0x030 HandleContentionEvent : _EX_PUSH_LOCK

   +0x038 DebugInfo        : (null)

   +0x040 ExtraInfoPages   : 0n0

   +0x044 Flags            : 0

   +0x044 StrictFIFO       : 0y0

   +0x048 FirstFreeHandle  : 0x10004

   +0x050 LastFreeHandleEntry : 0xfffff8a0`10ca4ff0 _HANDLE_TABLE_ENTRY

   +0x058 HandleCount      : 0x3fc0

   +0x05c NextHandleNeedingPool : 0x10400

   +0x060 HandleCountHighWatermark : 0x3fc1

 

The FirstFreeHandle contains the handle number that will be given to the next handle allocated from this table.  This value is encoded, to get the next handle number we need to right shift the FirstFreeHandle by 2 bits.

 

2: kd> ?00010004>>2

Evaluate expression: 16385 = 00000000`00004001

 

The result from above, 0x4001, is greater than the number of possible string atoms.  As I mentioned earlier, there is a limit of 0x4000 string atoms.  Now we know that the session is out of string atoms.

 

The next step is to dump the string atoms to identify whether there is an observable pattern in the leaked strings.  The !atom command only works in user mode, so we need to dump the kernel mode strings manually.  An atom table is comprised of multiple buckets.   Each bucket is the head of a list of atoms.  The buckets start at offset 0x20 in the atom table in 64-bit, and offset 0x10 in 32-bit.

 

2: kd> dq fffff8a0`05e5bc70+20

fffff8a0`05e5bc90  fffff8a0`05e5ba60 fffff8a0`05db7be0

fffff8a0`05e5bca0  fffff8a0`08cf1770 fffff8a0`05e5b3d0

fffff8a0`05e5bcb0  fffff8a0`05ea9020 fffff8a0`05e5b8e0

fffff8a0`05e5bcc0  fffff8a0`05ea9b10 fffff8a0`05ea9910

fffff8a0`05e5bcd0  fffff8a0`05ea9f00 fffff8a0`05e5b650

fffff8a0`05e5bce0  fffff8a0`05cda290 fffff8a0`05ea9e80

fffff8a0`05e5bcf0  fffff8a0`05e5b200 fffff8a0`05ea9e30

fffff8a0`05e5bd00  fffff8a0`05e5b7e0 fffff8a0`06c56210

2: kd> dq

fffff8a0`05e5bd10  fffff8a0`06d6b5a0 fffff8a0`05ea9d50

fffff8a0`05e5bd20  fffff8a0`05e5b790 fffff8a0`05e5b9d0

fffff8a0`05e5bd30  fffff8a0`06bd9bc0 fffff8a0`05ea9c90

fffff8a0`05e5bd40  fffff8a0`05e5b0c0 fffff8a0`06ae2020

fffff8a0`05e5bd50  fffff8a0`05e5b930 fffff8a0`04d2af40

fffff8a0`05e5bd60  fffff8a0`05e5b690 fffff8a0`05e5b980

fffff8a0`05e5bd70  fffff8a0`05e5b490 fffff8a0`05e5b410

fffff8a0`05e5bd80  fffff8a0`05e5ba20 fffff8a0`05e5b4f0

2: kd> dq

fffff8a0`05e5bd90  fffff8a0`05e5baa0 fffff8a0`05e5b390

fffff8a0`05e5bda0  fffff8a0`05e5b840 fffff8a0`05ea9c50

fffff8a0`05e5bdb0  fffff8a0`05e5b250 00000000`00000000

fffff8a0`05e5bdc0  00000000`00000000 00000000`00000000

fffff8a0`05e5bdd0  00000000`00000000 00000000`00000000

fffff8a0`05e5bde0  00000000`00000000 00000000`00000000

fffff8a0`05e5bdf0  00000000`00000000 00000000`00000000

fffff8a0`05e5be00  00000000`00000000 00000000`00000000

 

The quick and dirty way to dump the buckets is with !list.  I am sure that some will say it is tedious to dump each bucket list by hand and that there are easier ways to accomplish this.  To prevent this article from becoming a lesson on debugger scripting, I am leaving that as an exercise to the reader.

 

2: kd> !list "-t nt!_RTL_ATOM_TABLE_ENTRY.HashLink -e -x \"du @$extret+10\" fffff8a0`05e5ba60"

du @$extret+10

fffff8a0`05e5ba70  "Native"

 

<snip strings that don't match a pattern>

 

du @$extret+10

fffff8a0`0838a120  "ControlOfs0210000000000700"

 

du @$extret+10

fffff8a0`0f7ff430  "ControlOfs021A000000000C30"

 

du @$extret+10

fffff8a0`162168c0  "ControlOfs020E000000001774"

 

du @$extret+10

fffff8a0`08c33870  "ControlOfs01F70000000007F4"

 

du @$extret+10

fffff8a0`07c46910  "ControlOfs0202000000000BF8"

 

du @$extret+10

fffff8a0`062aab50  "ControlOfs01F5000000001274"

 

du @$extret+10

fffff8a0`0777b150  "ControlOfs0202000000000C80"

 

du @$extret+10

fffff8a0`07dd3410  "ControlOfs0207000000000F00"

 

du @$extret+10

fffff8a0`0f01d190  "ControlOfs0214000000000DAC"

 

Dumping the atoms I found that there is a continuous pattern of the string ControlOfs followed by 16 hexadecimal numbers.  Some time spent with your favorite search engine should find other reports of atom leaks involving the string ControlOfs, and that these leaks have been identified as a problem in some specific software.  In this instance the programmer using that software needs to change their application to avoid the problem.

Comments

• Anonymous
  January 31, 2012
  The comment has been removed

• Anonymous
  February 01, 2012
  The comment has been removed

• Anonymous
  February 02, 2012
  I think you forgot to mention that RegisterWindowMessages is as well including an atom which is never released. That is why atom is being depleted so fast when we call RWM several times. Do you know then how to delete the atom leaked when using RWM? [Jordi, RegisterWindowMessage does not provide a mechanism to unregister the message.  The Windows Development Reference for RegisterWindowMessage does not directly say that it can exhaust the global atom table; however, it does inform a programmer that there are a limited number of message identifiers and that the message remains registered until the session ends.  Samples which include this API use a pre-defined string, which increments a reference count rather than continuously creating new atoms.]

• Anonymous
  February 02, 2012
  When the atom table is depleted it returns the error "System Error. Code: 8. Not enough storage is available to process this command".

• Anonymous
  February 14, 2012
  Do you know if CreateProcess (msdn.microsoft.com/.../ms682425(v=vs.85).aspx) and CoCreateInstance (msdn.microsoft.com/.../ms686615(v=vs.85).aspx) functions generate atoms as well?. [Hi Jordi.  CreateProcess does not create an atom from the global atom table discussed here, although the process which is created may do so.  CoCreateInstance doesn't use atoms from the table discussed in this article, although ITfCategoryMgr may use a different type of atom.]

• Anonymous
  March 12, 2012
  Hi There - I am debugging an issue with global atom leaks.  I have a dump on our test computers that has an extremely similar set of leaked global atoms.  I followed the procedure in this article and dumped out thousands of ControlOfs atoms.  All signs on the Iternet point to this as bug in Delphi projects.  However, we are using vanilla MSVC + QT.  I can't find any reference to controLOfs in our source base, the QT source base, or any of the binaries we use.  Can you shed any light in the third party software that causes this problem? Thanks! [I am only aware of ControlOfs atoms being allocated by the source you mentioned.  Keep in mind that atoms may be allocated by any software running on the system, not necessarily your application.  If you look at some of the documetation around the use of ControlOfs and atoms, you'll see that one of the numbers is the PID and one is the TID.  If you track PIDs over time you may be able to match up a PID to a process name.  One way to do this is to audit process tracking, and view the event 592 messages to find the PID you are interested in.] [UPDATE: Sorry, I was wrong about the ControlOfs atom containing the PID.  It contains the TID. You can use procmon with a filter on ThreadCreate to get a historical list of TIDs.  There are likely other methods to get a list of thread IDs, procmon is the first one that came to mind.]

• Anonymous
  November 26, 2015
  Hi, while looking for a similar problem, I found this article. I have thousends of such entries that finally make my Windows 7 unuseable: E363 = TitleBarWindow:dc602614-2cbc-4963-a0e3-fbded66b58c9  --RWM E364 = TitleBarWindow:e6f45d8d-9a54-441b-ac83-d37416f9276d  --RWM E365 = TitleBarWindow:57ad0825-fb28-46bf-a19d-4d1a5a72276f  --RWM E366 = TitleBarWindow:b4d8e325-5bf3-49e6-b124-db72512f89b8  --RWM E367 = TitleBarWindow:69219ce6-31eb-4a48-b2db-f81141038d97  --RWM E368 = TitleBarWindow:caf86044-aa57-40da-91c1-ad62c8bad662  --RWM E369 = TitleBarWindow:7a0c3c9c-c826-4ab5-8355-759f8ae895f5  --RWM I have no idea which application or Windows API function creates these entries. Google or bing search does not show any significant hits either. Do you have any idea? Regards, Andreas   [Those strings do not resemble an atom leak we have seen before.  This article describes several APIs used to allocate global atoms, the best option is probably to set some breakpoints.]