Share via

Work Queues and Dispatcher Headers

  • Article

 

Introduction

Hi everyone, Bob here again with a description of Work Queues and Dispatcher Headers. For those of you that look at dumps, you may have noticed that there are always threads waiting at KeRemoveQueue. You may have wondered what this function does. Well, I’m glad you asked… J

What are those threads doing?

Those threads waiting on the Remove Queue are worker threads. Worker threads are used when a system task cannot or does not want to do a particular task. For example, a thread running a DPC cannot pend and wait for a task to be done, so it sends the work to a worker thread.

How does this mechanism work?

The worker thread and the entities that are going to give the worker thread work, each know of a KQUEUE structure. The KQUEUE structure is initialized and, since the queue has an embedded dispatcher object, the worker thread pends on it waiting to be signaled. That is what you see on one of these waiting stacks.

Below is a KQUEUE:

typedef struct _KQUEUE {
DISPATCHER_HEADER Header;
LIST_ENTRY EntryListHead;
ULONG CurrentCount;
ULONG MaximumCount;
LIST_ENTRY ThreadListHead;
} KQUEUE, *PKQUEUE, *RESTRICTED_POINTER PRKQUEUE;

Below is an example of a waiter:

Priority 9 BasePriority 9 PriorityDecrement 0

 Child-SP RetAddr Call Site

 fffffadc`b053dab0 fffff800`01027752 nt!KiSwapContext+0x85

 fffffadc`b053dc30 fffff800`01024ef0 nt!KiSwapThread+0x3c9 ß Waits on the dispatcher object

 fffffadc`b053dc90 fffffadc`b9a380b0 nt!KeRemoveQueue+0x656

 fffffadc`b053dd10 fffff800`0124b972 srv!WorkerThread+0xb0

 fffffadc`b053dd70 fffff800`010202d6 nt!PspSystemThreadStartup+0x3e

 fffffadc`b053ddd0 00000000`00000000 nt!KxStartSystemThread+0x16

What is a dispatcher object?

A dispatcher object can be passed into kernel routines such as KeWaitForSingleObject. This object is a synchronization object. This means that a thread can wait on this object until another thread “signals” it. The function KeRemoveQueue is waiting for its dispatcher object to be “signaled”.

Below is a dispatcher object. Basically threads are queued on this object until the object is “signaled”. Once that happens the waiting thread is readied for execution.

nt!_DISPATCHER_HEADER

   +0x000 Type : UChar

   +0x001 Absolute : UChar

   +0x001 NpxIrql : UChar

   +0x002 Size : UChar

   +0x002 Hand : UChar

   +0x003 Inserted : UChar

   +0x003 DebugActive : UChar

   +0x000 Lock : Int4B

   +0x004 SignalState : Int4B ßSet when the object is signaled.  

   +0x008 WaitListHead : _LIST_ENTRY ßList of waiters on this object.

Below is an actual dispatcher object for a queue:

5: kd> dt nt!_dispatcher_header fffffadcdb3ed368

nt!_DISPATCHER_HEADER

   +0x000 Type : 0x4 ''

   +0x001 Absolute : 0 ''

   +0x001 NpxIrql : 0 ''

   +0x002 Size : 0x10 ''

   +0x002 Hand : 0x10 ''

   +0x003 Inserted : 0 ''

   +0x003 DebugActive : 0 ''

   +0x000 Lock : 1048580

   +0x004 SignalState : 0

   +0x008 WaitListHead : _LIST_ENTRY [ 0xfffffadc`db3f4ce8 - 0xfffffadc`da74dce8 ] ß List of threads waiting for this object

Each thread has a Wait List entry for each object it is waiting for:

5: kd> dt nt!_KWAIT_BLOCK 0xfffffadc`db3f4ce8

   +0x000 WaitListEntry : _LIST_ENTRY [ 0xfffffadc`da74dce8 - 0xfffffadc`db3ed370 ] ßNext thread waiting for this object

   +0x010 Thread : 0xfffffadc`db3f4bf0 _KTHREAD ßThe thread waiting for the object

   +0x018 Object : 0xfffffadc`db3ed368 ßThe object the thread is waiting for (queue object)

   +0x020 NextWaitBlock : 0xfffffadc`db3f4ce8 _KWAIT_BLOCK ßNext object this thread is waiting for (thread 0xfffffadc`db3f4bf0) if any.

   +0x028 WaitKey : 0

   +0x02a WaitType : 0x1 ''

   +0x02b SpareByte : 0 ''

   +0x02c SpareLong : 1533340

What wakes up or signals the thread?

When the thread is waiting, an entity can call KeInsertQueue to insert elements in the work queue. When that event happens the thread is woken up and the system will remove the entry from the work queue and the call from KeRemoveQueue will return with the element. If the thread is not waiting when the call is made, the dispatcher object is put in the queue and the next call to KeRemoveQueue will not pend.

What about synchronization objects?

When one thread wants to synchronize with another, a synchronization object (such as an event) is used. When a thread waits for an event, another thread will signal the event when a job is done such as I/O. The dispatcher objects above are used for all the synchronization objects. As you can see by how the structures are designed, one thread can wait for many objects.

Below this thread is waiting for a synchronization object.

 

THREAD fffffadff752b040 Cid 0004.2858

    fffffadcbe1c3768 NotificationEvent ßObject thread is waiting for .

Not impersonating

DeviceMap fffffa80000840f0

Owning Process fffffadce06e15a0 Image: System

Wait Start TickCount 49664324 Ticks: 247591 (0:01:04:28.609)

Context Switch Count 1

UserTime 00:00:00.000

KernelTime 00:00:00.000

Start Address EmcpBase (0xfffffadcbe22d810)

Stack Init fffffadcb870be00 Current fffffadcb870b940

Base fffffadcb870c000 Limit fffffadcb8706000 Call 0

Priority 8 BasePriority 8 PriorityDecrement 0

Child-SP RetAddr : Args to Child : Call Site

fffffadc`b870b980 fffff800`01027752 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSwapContext+0x85

fffffadc`b870bb00 fffff800`0102835e : 00000000`00000000 00000000`00000000 fffffadf`f752b0d8 fffffadf`f752b040 : nt!KiSwapThread+0x3c9

fffffadc`b870bb60 fffffadc`be21832b : 00000000`00000000 fffff800`00000000 00000000`00000000 fffffadc`be88b100 : nt!KeWaitForSingleObject+0x5a6

fffffadc`b870bbe0 fffffadc`be1bd0da : 00000000`00000004 00000000`00000000 fffffadc`be239c40 00000000`00000000 : EmcpBase+0xb32b

fffffadc`b870bc20 fffffadc`be22c9a1 : 00000000`00000000 fffffadc`b870bd08 fffffadc`be239c40 fffffadc`e06f6fe0 : EmcpMPAA+0xd0da

fffffadc`b870bc70 fffffadc`be22d90b : fffffadc`da2338c0 00000000`00000001 fffffadc`d9eb3c10 fffffadc`b870bd08 : EmcpBase+0x1f9a1

fffffadc`b870bce0 fffff800`0124b972 : fffffadc`d9f85780 fffffadf`f752b040 00000000`00000080 fffffadf`f752b040 : EmcpBase+0x2090b

fffffadc`b870bd70 fffff800`010202d6 : fffff800`011b1180 fffffadf`f752b040 fffff800`011b5500 00000000`00000000 : nt!PspSystemThreadStartup+0x3e

fffffadc`b870bdd0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KxStartSystemThread+0x16

Dispatcher header from address above:

 

5: kd> dt nt!_dispatcher_header fffffadcbe1c3768

nt!_DISPATCHER_HEADER

   +0x000 Type : 0 ''

   +0x001 Absolute : 0 ''

   +0x001 NpxIrql : 0 ''

   +0x002 Size : 0x6 ''

   +0x002 Hand : 0x6 ''

   +0x003 Inserted : 0 ''

   +0x003 DebugActive : 0 ''

   +0x000 Lock : 393216

   +0x004 SignalState : 0

   +0x008 WaitListHead : _LIST_ENTRY [ 0xfffffadf`f752b138 - 0xfffffadf`f752b138 ]

Wait block for this thread:

5: kd> dt 0xfffffadf`f752b138 _KWAIT_BLOCK

nt!_KWAIT_BLOCK

   +0x000 WaitListEntry : _LIST_ENTRY [ 0xfffffadc`be1c3770 - 0xfffffadc`be1c3770 ]

   +0x010 Thread : 0xfffffadf`f752b040 _KTHREAD

   +0x018 Object : 0xfffffadc`be1c3768

   +0x020 NextWaitBlock : 0xfffffadf`f752b138 _KWAIT_BLOCK

   +0x028 WaitKey : 0

   +0x02a WaitType : 0x1 ''

   +0x02b SpareByte : 0 ''

   +0x02c SpareLong : 1

Conclusion

I hope this gives a better understanding of Work Queues and Dispatcher Headers. More detailed information can be found here: https://msdn2.microsoft.com/en-us/library/ms810047.aspx and here:  https://www.microsoft.com/whdc/driver/kernel/locks.mspx.

Comments