Signaling a CPU event from a kernel-mode driver

Article
09/14/2022

There are cases when the kernel-mode driver (KMD) needs to signal a CPU event to notify the user-mode driver (UMD) about something; for example:

When KMD detects that one of its objects is in a bad state and needs to notify UMD.
During GPU debugging where KMD needs to communicate to UMD that some event happened. For IHVs with Control Panels for GPU, signaling CPU events by KMD allows KMD to notify the Control Panels app about internal events.

Typically, UMD can create a CPU event and pass its NT handle to KMD in an escape private data. This method does not work in the GPU paravirtualization (GPU-PV) scenario because NT handles cannot be used across virtual machine boundaries.

Starting in Windows 11 version 21H2 (WDDM 3.0), the WDDM API was extended to allow UMD to create a CPU event object that can be signaled by KMD. This feature works both when UMD is running on the host or in a virtual machine using GPU-PV.

Feature flow

UMD creates a CPU event.
UMD creates a GPU synchronization object with the D3DDDI_CPU_NOTIFICATION type. The created object is made visible to KMD by setting the SignalByKmd flag when calling D3DKMTCreateSynchronizationObject.
Dxgkrnl calls DXGKDDI_CREATECPUEVENT to allow KMD to create its own object.
UMD calls D3DKMTEscape with the D3DDDI_DRIVERESCAPETYPE_CPUEVENTUSAGE known escape type to notify KMD about intended usage of the synchronization object.
Dxgkrnl calls DXGKDDI_ESCAPE to pass the private data to KMD.
At some point KMD calls DXGKCB_SIGNALEVENT with the CpuEventObject flag to signal the CPU event object.
UMD calls D3DKMTDestroySynchronizationObject to destroy the CPU event object.
Dxgkrnl calls DXGKDDI_DESTROYCPUEVENT to destroy the CPU event object. DXGKCB_SIGNALEVENT should not be called after this point.

The synchronization object cannot be inserted to a context queue. It can only be signaled by KMD using DXGKCB_SIGNALEVENT.

User-mode APIs to handle CPU event sync objects

Creation of the KMD CPU event object

The KMD CPU event object is created as a GPU synchronization object by calling D3D12DDICB_CREATESYNCHRONIZATIONOBJECT2 with:

Type set to D3DDDI_CPU_NOTIFICATION.
Flags set to SignalByKmd to specify that the object will be signaled by KMD. This flag can be set only when the Type member ofD3DDDI_SYNCHRONIZATIONOBJECTINFO2 is D3DDDI_CPU_NOTIFICATION.

When the SignalByKmd flag is set, DXGKDDI_CREATECPUEVENT will be called to create the KMD CPU event object. Note that the device handle must be specified when creating the synchronization object.

The synchronization object cannot be used in signal and wait APIs (D3DKMTSignalSynchronizationObject, D3DKMTWaitForSynchronizatioObject). It can be signaled only by KMD, and UMD can wait on the corresponding CPU event.

UMD escape to define the usage for a KMD CPU event sync object

A known escape was added to D3DDDI_DRIVERESCAPETYPE. D3DDDI_DRIVERESCAPETYPE_CPUEVENTUSAGE is used to notify KMD about the intended usage of a KMD CPU event object. A known escape is defined by setting DXGKARG_ESCAPE::Flags.DriverKnownEscape = 1. Known escapes are sent to the host even from secure virtual machines.

The following snippet is a usage example.

D3DDDI_DRIVERESCAPE_CPUEVENTUSAGE Command = {};
Command.EscapeType = D3DDDI_DRIVERESCAPETYPE_CPUEVENTUSAGE;
Command.hSyncObject = SyncObjectHandle;
Command.Usage[0] = 1;

D3DKMT_ESCAPE Args = {};
Args.hAdapter = AdapterHandle;
Args.Type = D3DKMT_ESCAPE_DRIVERPRIVATE;
Args.Flags.DriverKnownEscape = 1;
Args.Flags.NoAdapterSynchronization = 1; // Prevent waking up the device from D3
Args.pPrivateDriverData = &Command;
Args.PrivateDriverDataSize = sizeof(Command);

NTSTATUS Status = D3DKMTEscape(&Args);

Dxgkrnl will call DXGKDDI_ESCAPE with the following:

hDevice set to the miniport device handle that was used to create the sync object
pPrivateDriverData pointing to a D3DDDI_DRIVERESCAPE_CPUEVENTUSAGE structure
PrivateDriverDataSize set to sizeof(D3DDDI_DRIVERESCAPE_CPUEVENTUSAGE)

Creating and destroying a KMD CPU event object

The following DDIs are used to create and destroy KMD CPU event sync objects:

Signaling a CPU event object from KMD

To signal a CPU event object, KMD calls DXGKCB_SIGNALEVENT at IRQL <= DISPATCH_LEVEL and with the DXGKARGCB_SIGNALEVENT structure values set as follows:

hDxgkProcess equals 0.
hEvent equal to the Dxgkrnl CPU event object handle passed in to DXGKDDI_CREATECPUEVENT.
CpuEventObject must be 1.
Reserved must be 0.

Share via