The Good, the bad, and the ugly: A 3 part series about Windows Event Log schemas

One question that comes up frequently is a frustration with event schemas. Common statements I've heard include:

1. "Windows Events have no set schema!"
2. I need to know the OS version to know what the event schema
3. There's only a list of events for the Security channel.

As you probably guessed from the title, there are three scenarios for how event schemas are defined. I'll explain the different definition methods, their benefits and limitations, and provide some utilities to for you to use to collect the event definitions for individual event analysis or use in your own automation.

Event Log Schema overview

Before we get into a specific event or event provider/event source's events, lets talk a bit about the event log format. Starting in Windows Vista, the event log was completely re-written (though it includes backwards compatibility) down to the file format - it wasn't just adding an "X" to the file extension for the log file (previously was EVT, now is EVTX.) Practically everything you knew about the event log changed but if you were a developer you didn't have to change due to backwards compatibility in place. I'll cover the differences in more detail in future posts.

One change was an XML schema being defined for event format. The schema is publically available in the Windows SDK, you'll find it under the SDK path under [OS Version]/include/um/Event.xsd - it shows the required fields for an event. This has the definition for not only Event Log events but also Event Tracing for Windows [ETW] events, since the API between the Event Log and ETW was unified in Vista. However, don't read it expecting to find the schema for every possible event, it's just the rules about a general schema for event structure. Each event provider/event source defines the detailed event schema.

Long story short, events in the Event Log will have a <System> element and either a <EventData> or <UserData> element - those last two are the event payload, which include specific fields and values for the event.

N.B: When I use "Event Provider" that will refer to a modern (Vista+), manifested event provider, which is covered below. An "Event Source" will refer to a legacy (pre-Vista) event source.

What defines an event?

When you talk to most people, they focus on the Event ID alone. In reality, event ID is not globally unique across the event log or even within a single event channel. You need to include the Event Provider (formerly known as the Event Source) and the Event ID to define an event.

Manifested providers, introduced in Windows Vista and covered below, go one step further - the event has a version for the payload (the <EventData> or <UserData> element) so that for a modern event provider you need to specify the Provider + EventID + Version to determine the event schema. All of this is included in the event <System> element so just by looking at those fields you can determine what the event schema should be.

The Good: Manifested event providers

A manifested event provider starts with an Instrumentation Manifest, which includes a list of events, event keywords, event descriptions, destination event channels, and event templates that the application will generate.

While manifests were not introduced in starting in Vista but the new instrumentation manifest centralized all of an event provider's information in one place in a standard format. Previously this event provider information did have a format but it was not enforced and therefore different developers implemented their own mechanism for storing keywords, channels, event descriptions in an event source's resource DLL. In short, it was a bit of a mess.

While the new method is not perfect, it does help you (the analyst or operator) have a single place to go for an event's information. Now if you have a modern event provider you're interested in, you can query for all events, their descriptions, event templates, event levels (information/warning/Error) quite easily.

Event Providers also turned on aspect of the legacy Event Source on its head. For an Event Source, you registered the event source for a single event log. A Manifested Event Provider doesn't do that - it is event log agnostic. Meaning, that at the event level the developer can specify what event log the event will go to. Fortunately that is defined in the event manifest so it's not a mystery where events are being logged.

Let's start with an example: Say you're interested in Security Audit events. You can use the PowerShell Get-WinEvent cmdLet to get the Security provider information. First you need the provider name, it's not the event log name "Security" so - where do you get the provider name?

It's actually in the first field of the event <System> element, here are the first few lines of an event:

<Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Security-Auditing" Guid=" {54849625-5478-4994-A5BA-3E3B0328C30D} " /><EventID>4688</EventID><Version>2</Version><Level>0</Level>

Etc…

The Name value is what you want to use for with Get-WinEvent:

[Note - None of the following commands need to be run from an elevated PowerShell session. You're just looking at event provider data, not the actual events itself. Unless I specifically call it out, you cannot change anything by running these commands.]
PS C:\> Get-WinEvent -ListProvider Microsoft-Windows-Security-Auditing Name     : Microsoft-Windows-Security-Auditing LogLinks : {Security} Opcodes : {win:Info} Tasks   : {SE_ADT_SYSTEM_SECURITYSTATECHANGE, SE_ADT_SYSTEM_SECURITYSUBSYSTEMEXTENSION, SE_ADT_SYSTEM_INTEGRITY, SE_ADT_SYSTEM_IPSECDRIVEREVENTS...} PS C:\>

In itself, not very interesting but the detail is one level deeper, specifically under the Events property.
PS C:\> $SecEvents = Get-WinEvent -ListProvider Microsoft-Windows-Security-Auditing PS C:\> ## get count of events for the provider PS C:\> $SecEvents.Events.Count 443
Now we know there are 443 events for the "Microsoft-Windows-Security-Auditing" provider.
If you're interested in the Process Create event generated by the Microsoft-Windows-Security-Auditing provider (I abbreviate events like this: Security/4668 or where I need to be completely unambiguous: Security/4688/2 to include the event schema version.)

PS C:\> $SecEvents.Events | where-object {$_.ID -eq "4688"}
[Wall of text omitted]

 

On a Windows 10 system, this will return 3 event definitions. While three are defined, only one is actually generated. Actually you don't need to know what OS the event came from, it tells you the version of the event definition, right below the Event ID.

Here's the command to get the schema for the Process Create event that matches the version above.
PS C:\> $SecEvents.Events | where-object {$_.ID -eq "4688" -and $_.Version -eq "2"}

Id         : 4688
Version     : 2
LogLink     : System.Diagnostics.Eventing.Reader.EventLogLink
Level       : System.Diagnostics.Eventing.Reader.EventLevel
Opcode     : System.Diagnostics.Eventing.Reader.EventOpcode
Task       : System.Diagnostics.Eventing.Reader.EventTask
Keywords   : {}
Template   : <template xmlns="https://schemas.microsoft.com/win/2004/08/events">
<data name="SubjectUserSid" inType="win:SID" outType="xs:string"/>
<data name="SubjectUserName" inType="win:UnicodeString" outType="xs:string"/>
<data name="SubjectDomainName" inType="win:UnicodeString" outType="xs:string"/>
<data name="SubjectLogonId" inType="win:HexInt64" outType="win:HexInt64"/>
<data name="NewProcessId" inType="win:Pointer" outType="win:HexInt64"/>
<data name="NewProcessName" inType="win:UnicodeString" outType="xs:string"/>
<data name="TokenElevationType" inType="win:UnicodeString" outType="xs:string"/>
<data name="ProcessId" inType="win:Pointer" outType="win:HexInt64"/>
<data name="CommandLine" inType="win:UnicodeString" outType="xs:string"/>
<data name="TargetUserSid" inType="win:SID" outType="xs:string"/>
<data name="TargetUserName" inType="win:UnicodeString" outType="xs:string"/>
<data name="TargetDomainName" inType="win:UnicodeString" outType="xs:string"/>
<data name="TargetLogonId" inType="win:HexInt64" outType="win:HexInt64"/>
<data name="ParentProcessName" inType="win:UnicodeString" outType="xs:string"/>
<data name="MandatoryLabel" inType="win:SID" outType="xs:string"/>
</template>

Description : A new process has been created.
Creator Subject:
Security ID:           %1
Account Name:           %2
Account Domain:         %3
Logon ID:               %4

Target Subject:
Security ID:           %10
Account Name:           %11
Account Domain:         %12
Logon ID:               %13

Process Information:
New Process ID:         %5
New Process Name:       %6!S!
Token Elevation Type:   %7
Mandatory Label:               %15
Creator Process ID:     %8
Creator Process Name:   %14!S!
Process Command Line:   %9!S!

Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account
Control policy.

Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account
Control is disabled or if the user is the built-in Administrator account or a service account.

Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User
Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is
also used when an application is configured to always require administrative privilege or to always require maximum
privilege, and the user is a member of the Administrators group.

Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited
token is used when User Account Control is enabled, the application does not require administrative privilege, and the
user does not choose to start the program using Run as administrator.

The Template property is the definition for the event's <EventData> element.
The Description property is the text message that will be displayed in the Event Viewer when the event is rendered, and the numbers are the replacement strings used by the EvtFormatMessage function to display the text.

For a decoder-ring for the symbols in the Description property (for example, the above event has !S! in a few replacement values.) Look at the Remarks section under FormatMessage.

There you have it, there's a start of how to get an event's schema, at least for manifested event providers.

I've written some PowerShell scripts to enumerate all events on a system and generate a file that can be imported into easily into Excel or SQL or used as a Hadoop mapping file.

Check it out here: https://github.com/tedhardyMSFT/EventLogUtilities - the manifested providers is the Get-ProviderEventMessages.ps1 file.

 

Next up, Legacy Event Sources!