Share via

Kerberos PAC Validation… what is it?

  • Article

 

 

Kerberos is well… Kerberos. And NTLM is NTLM right? Right. Most of the time. This post is from a recent hotfix I worked on where it was made painfully clear that this isn’t always true.

 

Here is a cool overview stolen from msdn

 

 

 

And the following table snipped:

 

Msv1_0.dll

The NTLM authentication protocol. This protocol authenticates clients that do not use Kerberos authentication.

Kerberos.dll

The Kerberos V5 authentication protocol. This protocol provides authentication using Kerberos protocol instead of plaintext, NTLM, or digest method.

 

Most people think that Kerberos.dll and msv1_0.dll never really interact. However there is one very important interaction which slips by people until it bites them in the rear. PAC’s.

 

First , what is the PAC ? PAC stands for Privilege Attribute Certificate

I won’t go into gory detail here but let’s say that the PAC contains various types of authorization data including groups that the user is a member of, rights the user has, and what policies apply to the user. When the client receives a ticket, the information contained in the PAC is used to generate the user’s access token. See https://msdn2.microsoft.com/en-us/library/aa302203.aspx for more info.

 

So how does this PAC thing work?

 

Well the Kerb client basically gets a ticket and then needs to do what is called PAC verification on the information ( to make sure it’s all cool to move ahead, after all if you read whats in it.. it’s pretty important ) We talk a little about it here https://support.microsoft.com/kb/906736

 

 

In order to do this we pass the information over and through the NTLM provider, msv1_0.dll and from there over the netlogon secure channel. Ahh it always seems to come back to it’s roots eh? J

 

You will see this in the netlogon logs as:

 

10/30 08:10:29 [LOGON] SamLogon: Generic logon of DOMAIN.COM\(null) from (null) Package:Kerberos Returns 0x0

 

 

Note the Kerberos package used over netlogon. Faskinating.

 

Anyway… how interesting, but lets apply it to some scenario.

 

The Perfect Storm

 

 

Let’s say you have assigned a number of application to a machine via GPO’s. You are concerned about licenses for these applications so when you remove machines you want the applications to be removed and checked the box you see below for “Uninstall this application when it falls out of the scope of management”

 

 

One day you find all the applications uninstalled on random machines ( one of them is the CIO ). Every single application. Uninstalled, even though you have not changed the policy or removed the machine from scope of the GPO.

 

You are a curious admin and prefer to keep your job. So.. you look at the userenv logs:

 

You see this:

USERENV(370.8fc) 16:13:11:240 ProcessGPOs: -----------------------

USERENV(370.8fc) 16:13:11:240 ProcessGPOs: Processing extension Software Installation

USERENV(370.8fc) 16:13:11:240 ReadStatus: Read Extension's Previous status successfully.

USERENV(370.8fc) 16:13:11:240 CheckForGPOsToRemove: GPO < Line of Business Applications-1> needs to be removed

USERENV(370.8fc) 16:13:11:240 CheckForGPOsToRemove: GPO <Line of Business Applications-2> needs to be removed

USERENV(370.8fc) 16:13:11:240 CheckForGPOsToRemove: GPO <Line of Business Applications> needs to be removed

USERENV(370.8fc) 16:13:11:250 CheckForGPOsToRemove: GPO <Very Important Apps> needs to be removed

 USERENV(370.8fc) 16:13:11:250 GetDeletedGPOList: Finished.

USERENV(370.8fc) 16:13:11:250 ProcessGPOList: Entering for extension Software Installation

USERENV(370.8fc) 16:13:11:250 MachinePolicyCallback: Setting status UI to Applying Software Installation policy...

USERENV(370.8fc) 16:13:11:300 LogExtSessionStatus: Successfully logged Extension Session data

USERENV(370.8fc) 16:13:11:301 MachinePolicyCallback: Setting status UI to Removing managed software Microsoft Project 2000…

 

 

Woah.

 

You open the event logs and look around that same time noted in the userenv logs..

 

Event Type: Error

Event Source: Kerberos

Event Category: None

Event ID: 7

Date: 11/6/1921

Time: 4:13:01 PM

User: N/A

Computer: MyClient

Description:

The kerberos subsystem encountered a PAC verification failure. This indicates that the PAC from the client MyClient$ in realm DOMAIN.COM had a PAC which failed to verify or was modified. Contact your system administrator.

 

 

And

Event Type: Error

Event Source: NETLOGON

Event Category: None

Event ID: 5719

Date: 11/6/1921

Time: 4:13:04 PM

User: N/A

Computer: MyClient

Description: No Domain Controller is available for domain DOMAIN due to the following:

                       The RPC server is unavailable.

                Make sure that the computer is connected to the network and try again.

 

And the corresponding netlogon log error:

 

11/01 13:14:07 [LOGON] SamLogon: Generic logon of DOMAIN.COM\(null) from (null) Package:Kerberos Returns 0xC000005E

 

And in the apps log:

 

Source: Application Management

Date: 11/6/9121 4:13:11 PM

Event ID: 303

Task Category: None

Level: Information

Keywords: Classic

User: SYSTEM

Computer: MyClient

Description:

The removal of the assignment of application Microsoft Project 2000 from policy LOB Applications succeeded.

 

 

 When systems encounter a Kerberos PAC validation error during log-on (perhaps due to transient network errors), it causes a machine to fall out of scope for all group policies, and all assigned applications go to an unmanaged state and to get uninstalled.

 

If you enable application management logging you will see something like this:

 

 

Software installation extension has been called for foreground synchronous policy refresh.

The following policies are to be removed, flags are 1.

    < Very Important AppsGPO > (unique identifier {4AE890C9-F0E7-4041-94A6-750896D93A3A})

        System volume path = \\domain.com\SysVol\domain.com\Policies\{4AE890C9-F0E7-4041-94A6-750896D93A3A}\Machine

        Active Directory path = LDAP://CN=Machine,CN={4AE890C9-F0E7-4041-94A6-750896D93A3A},CN=Policies,CN=System,DC=domain,DC=com

Enumerating the managed applications which are currently applied to this user.

The following 1 managed applications are currently applied to this user.

    Microsoft Project 2000 from policy < Very Important AppsGPO > with state 511 and assign count 1.

Policy < Very Important AppsGPO > has been removed. All applications will be made unmanaged or removed.

No Active Directory path.

Calling Windows Installer to remove application Microsoft Project 2000 from policy < Very Important AppsGPO > .

Removing application Microsoft Project 2000 from the software installation database.

The removal of the assignment of application Microsoft Project 2000 from policy < Very Important AppsGPO > succeeded.

 

 

So now the machine is toast.

 

Why? From the Event ID 7 we can see the PAC validation failed. These were machine scope GPO’s and the machine then failed to access the GPO. Whent it is denied access to the GPO, it thinks it has dropped out of scope and removes the apps. Quick – install this fix. https://support.microsoft.com/?kbid=929624

 

Why did it fail the PAC validation in the first place? We never found out. They had a complex network and perhaps had “hiccups” in the physical layers but in the end we thought – lets make this whole scenario more robust regardless of the *real* root cause. That’s hard for me to say sometimes, since we almost always want to get to true root cause.

Anyway… have fun and be careful with your forks and knives.

Spatdsg

 

 

March 9, 2007

Added a few more notes:

Vista ( and apparently 2k3 SP2 _ has an option to not do PAC validation for services - ValidateKdcPacSignature ( i guess call PSS for more info on this )

Also - on XP and 2k3 we dont do PAC validation for ID's who are network service account , or who have the TCB privilege.

Comments

  • Anonymous
    March 09, 2007
    Hi, We also had encountered the same problem on one of the machine, and searching for those events on eventid.net, we tried some solution listed there. One of them was to set MTU of the network right. After setting the MTU it solved the problem.

  • Anonymous
    March 09, 2007
    Ah good information. Thanks!

  • Anonymous
    July 12, 2007
    The comment has been removed

  • Anonymous
    February 04, 2008
    Why are we doing a PAC verification for a computer account while doing a logon? I thought PAC verification is carried out only if a service is run as a user account and not with local system. Since the computer account would have the Tcbprivilege, why do we do a PAC validation?

  • Anonymous
    March 26, 2009
    I had been meaning to blog about this for a while, and recently was teaching a class when a friend of

  • Anonymous
    May 09, 2009
    Edgar's post on Open Specification blog: Understanding Microsoft Kerberos PAC Validation

  • Anonymous
    October 15, 2009
    The comment has been removed

  • Anonymous
    November 01, 2009
    Agreed! That's why we developed the hotfix in http://support.microsoft.com/?kbid=929624

  • Anonymous
    April 02, 2010
    I am experiencing the kerberos PAC validation issue with my database servers running MS SQL 2005. The SQl services account which is a domain account always failed to authenticate at the same  and then resume after all.

  • Anonymous
    April 28, 2010
    The comment has been removed

  • Anonymous
    July 25, 2010
    In order to do this we pass the information over and through the NTLM provider, msv1_0.dll and from there over the netlogon secure channel. Ahh it always seems to come back to it’s roots eh? J ++++++++ So does the above mean that PAC verification would fail in a wk8R2 forest/domain if i disable NTLM completely using NTLM blocker.

  • Anonymous
    August 02, 2010
    Interesting question - I don't think so since we are using msv1_0.dll for secure channel transport - not necessarily the NTLM authentication protocol.  But - I have not tested this. My bet? I think it would still work OK.