Part3: Troubleshooting Kerberos authentication and things to check when it fails

Article
02/02/2018

This is a continuation post of part1 and part2 of my "Integrated Windows Authentication blog series" and last one in this series where we are going to discuss about what we can do when Kerberos Authentication fails, how to detect it and correct it!

Let me start by mentioning this --> C:\Windows\System32\Wininet.dll file calls the InitializeSecurityContext function to build the Kerberos ticket.

The InitializeSecurityContext (Kerberos) function initiates the client side, outbound security context from a credential handle. The function is used to build a security context between the client application and a remote peer. InitializeSecurityContext (Kerberos) returns a token that the client must pass to the remote peer, which the peer in turn submits to the local security implementation through the AcceptSecurityContext (Kerberos) call. The token generated should be considered opaque by all callers.

Typically, the InitializeSecurityContext (Kerberos) function is called in a loop until a sufficient security context is established.

From <https://msdn.microsoft.com/en-us/library/aa375507(VS.85).aspx>

I have explained in my Part1 post about how the kerberos authentication works, I strongly recommend reading that first before troubleshooting so we know what we are doing.

So, when the Kerberos AUTHETICATION FAILS or maybe not yet, This is how you verify/check/resolve:

Firewall Configuration:

Open firewall ports to allow HTTP traffic in on default and non-default ports. Ensure clients can connect to Kerberos Ports on the Active Directory.

Before testing authentication, ensure clients can access the SharePoint Server web applications on the configured HTTP ports. In addition, ensure clients can authenticate with Active Directory and request Kerberos tickets from the KDC over the standard Kerberos ports.

Open firewall ports to allow HTTP traffic in on default and non-default ports:

Typically you have to configure the firewall on each front-end Web to allow incoming requests over ports TCP 80 and TCP 443. Open Windows Firewall with Advanced Security and browse to the following Inbound Rules:

World Wide Web Services (HTTP Traffic-In)
World Wide Web Services (HTTPS Traffic-In)

Make sure the appropriate ports are open in your environment. In our example, we access SharePoint Server over HTTP (port 80), so this rule was enabled.

In addition, we have to open the non-default port used in our example (TCP 5555). If you have web sites running on non-default ports, you also have to configure custom rules to allow HTTP traffic on those ports.

Ensure that clients can connect to Kerberos ports on the Active Directory role

To use Kerberos authentication, clients will have to request ticket granting tickets (TGT) and service tickets (ST) from the Key Distribution Center (KDC) over UDP or TCP port 88. By default, when you install the Active Directory Role in Windows Server 2008 and later, the role will configure the following incoming rules to allow this communication by default:

Kerberos Key Distribution Center – PCR (TCP-In)
Kerberos Key Distribution Center – PCR (UDP-In)
Kerberos Key Distribution Center (TCP-In)
Kerberos Key Distribution Center (UDP-In)

In your environment ensure these rules are enabled and that clients can connect to the KDC (domain controller) over port 88.

From <https://technet.microsoft.com/en-us/library/gg502602(v=office.14).aspx>

Test browser authentication:

After configuring Active Directory, DNS and SharePoint Server you can now test whether Kerberos authentication is configured correctly by browsing to your web applications. When testing in the browser, ensure the following conditions are met:

The test user is logged into a Windows 7 or higher computer joined to the domain that SharePoint Server is installed in, or is logged into a domain trusted by the SharePoint Server domain.
Integrated Windows authentication is enabled in the browser. Under Internet Options in the Advanced tab, make sure Enable Integrated Windows Authentication* is enabled in the Security section:

Local intranet is configured to automatically logon clients. Under Internet explorer option, in the Security tab, select Local Intranet and click the Custom level button. Scroll down and make sure that Automatic logon only in Intranet zone is selected.
Ensure that Automatically detect intranet network is selected in Internet options->Security->Intranet Zone->Sites.
If you are using fully qualified domain names to access the SharePoint Server web applications, ensure that the FQDNs are included in the intranet zone, either explicitly or by wildcard inclusion (for example, "*.vmlab.local").

The easiest way to determine if Kerberos authentication is being used is by logging into a test workstation and navigating to the web site in question. If the user isn't prompted for credentials and the site is rendered correctly, you can assume Integrated Windows authentication is working.

From <https://technet.microsoft.com/en-us/library/gg502602(v=office.14).aspx>

If any of the above is not set properly, then Delegation fails even when Kerberos authentication works:

Check Internet Explorer Zone used for the URL.

Kerberos delegation is only allowed for the « Intranet » and « Trusted Sites » zones (in other words, IE sets the ISC_REQ_DELEGATE flag when it calls InitializeSecurityContext only if the zone computed is « Intranet » or « Trusted Sites » )
The account used for the application pool identity must have the « Trusted for delegation » flag set
If delegation still fails, you may consider using DELEGCONFIG. This tool allows to diagnose - and resolve - dozens of issues preventing Kerberos authentication from working correctly. Check the following link for more details: https://blogs.iis.net/bretb/archive/2008/03/27/How-to-Use-DelegConfig.aspx

The next step is to determine if the negotiate protocol was used to negotiate Kerberos authentication as the authentication provider for the request. This can be done in the following ways:

SharePoint server Front-end Windows security logs

If Kerberos authentication is working correctly you will see Logon events in the security event logs on the front-end webs with event ID = 4624. In the general information for these events you should see the security ID being logged onto the computer and the Logon Process used, which should be Kerberos.

Event is here below:

Log Name: Security

Source: Microsoft-Windows-Security-Auditing

Date: 5/6/2018 10:27:48 PM

Event ID: 4624

Task Category: Logon

Level: Information

Keywords: Audit Success

User: N/A

Computer: SP.contoso.com

Description:

An account was successfully logged on.

Subject:

Security ID: NULL SID

Account Name: -

Account Domain: -

Logon ID: 0x0

Logon Type: 3

Impersonation Level: Impersonation

New Logon:

Security ID: contoso\spfarm

Account Name: spfarm

Account Domain: contoso

Logon ID: 0x15636

Logon GUID: {02398194-25d2-4852-4d60-026f36f6632a}

Process Information:

Process ID: 0x0

Process Name: -

Network Information:

Workstation Name:

Source Network Address: -

Source Port: -

Detailed Authentication Information:

Logon Process: Kerberos

Authentication Package: Kerberos

Transited Services: -

Package Name (NTLM only): -

Key Length: 0

Event Xml:

<Channel>Security</Channel>

<Computer>SP.contoso.com</Computer>

</System>

<Data Name="TargetUserName">spfarm</Data>

<Data Name="TargetDomainName">contoso</Data>

<Data Name="LogonProcessName">Kerberos</Data>

<Data Name="AuthenticationPackageName">Kerberos</Data>

</Data>

</EventData>

</Event>

We should see same event on KDC (Domain controller) too:

Check if the IIS server is actually configured to send the WWW-Authenticate: Negotiate header:

If IIS doesn't send this header, you'll need to use ADSUTIL to set the Negotiate header though the NTAuthenticationProviders metabase property (see https://support.microsoft.com/kb/215383)
Note: by default, the NTAuthenticationProviders property is not set which causes IIS to send both Negotiate and NTLM headers:

To check or set the property: Locate the directory that contains the Adsutil.vbs file. By default, this directory is C:\Inetpub\Adminscripts.

Use the following command to retrieve the current values for the NTAuthenticationProviders metabase property:
cscript adsutil.vbs get w3svc/WebSite/root/NTAuthenticationProviders
In this command, WebSite is a placeholder for the Web site ID number. The Web site ID number of the default Web site is 1.

For Example:

PS C:\Users\Administrator.contoso> cscript "C:\inetpub\AdminScripts\adsutil.vbs" get w3svc/1834077867/root/NTAuthenticat

ionProviders

Microsoft (R) Windows Script Host Version 5.8

NTAuthenticationProviders : (STRING) "Negotiate,NTLM"

If the command above does not return the string "Negotiate,NTLM," use the following command to enable the Negotiate process:
cscript adsutil.vbs set w3svc/WebSite/root/NTAuthenticationProviders "Negotiate,NTLM"

Example: cscript adsutil.vbs set w3svc/1834077867/root/NTAuthenticationProviders "Negotiate,NTLM"

Warning: If you edit the metabase incorrectly, you can cause serious problems that may require that you reinstall any product that uses the metabase. Microsoft cannot guarantee that problems that result if you incorrectly edit the metabase can be solved. Edit the metabase at your own risk.

Note: Always back up the metabase before you edit it.

From <https://support.microsoft.com/en-us/help/215383/how-to-configure-iis-to-support-both-the-kerberos-protocol-and-the-ntl>

Check if the client and server are on the same machine:

Kerberos is not enabled in this configuration and a hard coded loopback check will always force usage of NTLM in this scenario. Note that NTLM may also not work in this configuration (see https://support.microsoft.com/kb/896861 for more details).

Workarounds:

Set DisableStrictNameChecking to 1 And

Either

[Preferred method if NTLM is needed]Create "Multi-String Value" BackConnectionHostNames = "HostName of the site" at location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0

Create Dword "DisableLoopbackCheck" = 1 at location HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa

Duplicate or Wrong Service principal name set for Application Pool identity:

If a wrong SPN is set for the APP Pool identity or if there are Duplicate SPNs set, kerberos AUTHENTICATION fails because it cannot determine which account to use to Decrypt the Ticket.

We can use these SETSPN.EXE commands to check any SPN related issues:

SETSPN -L [Application Pool account] --> Will list all SPNs registered for that account.

SETSPN -X --> Will show any Duplicate SPNs

SETSPN -Q --> Can be used to Query for Specific or all SPNs.

SETSPN.EXE Usage: C:\Windows\system32\setspn.exe [modifiers switch] [accountname]

Where "accountname" can be the name or domain\name

of the target computer or user account

Edit Mode Switches:

-R = reset HOST ServicePrincipalName

Usage: setspn -R accountname

-S = add arbitrary SPN after verifying no duplicates exist

Usage: setspn -S SPN accountname

-D = delete arbitrary SPN

Usage: setspn -D SPN accountname

-L = list SPNs registered to target account

Usage: setspn [-L] accountname

Query Mode Switches:

-Q = query for existence of SPN

Usage: setspn -Q SPN

-X = search for duplicate SPNs

Usage: setspn -X

Some Example commands with outputs are as below:

PS C:\Users\Administrator.contoso> SetSPN -S HTTP/Kerberos.contoso.com:9090 Contoso\SPAppPool

Checking domain DC=contoso,DC=com

Registering ServicePrincipalNames for CN=SPAppPool,CN=Users,DC=contoso,DC=com

HTTP/Kerberos.contoso.com:9090

Updated object

PS C:\Users\Administrator.contoso> setspn -l contoso\SPAppPool

Registered ServicePrincipalNames for CN=SPAppPool,CN=Users,DC=contoso,DC=com:

HTTP/Kerberos.contoso.com:9090

HTTP/Kerberos:9090

HTTP/Kerberos.contoso.com

HTTP/Kerberos

PS C:\Users\Administrator.contoso> SetSPN -S HTTP/Kerberos Contoso\SPAppPool

Checking domain DC=contoso,DC=com

CN=SPAppPool,CN=Users,DC=contoso,DC=com

HTTP/Kerberos.contoso.com:9090

HTTP/Kerberos:9090

HTTP/Kerberos.contoso.com

HTTP/Kerberos

Duplicate SPN found, aborting operation!

PS C:\Users\Administrator.contoso> setspn -Q http/sp

Checking domain DC=contoso,DC=com

No such SPN found.

PS C:\Users\Administrator.contoso> setspn -Q */SQL

Checking domain DC=contoso,DC=com

CN=SQL,CN=Computers,DC=contoso,DC=com

MSSQLSvc/SQL.contoso.com:1433

MSSQLSvc/SQL.contoso.com

TERMSRV/SQL

TERMSRV/SQL.contoso.com

WSMAN/SQL

WSMAN/SQL.contoso.com

RestrictedKrbHost/SQL

HOST/SQL

RestrictedKrbHost/SQL.contoso.com

HOST/SQL.contoso.com

Existing SPN found!

PS C:\Users\Administrator.contoso> setspn -Q */sp

Checking domain DC=contoso,DC=com

CN=SP,CN=Computers,DC=contoso,DC=com

TERMSRV/SP

TERMSRV/SP.contoso.com

WSMAN/SP

WSMAN/SP.contoso.com

RestrictedKrbHost/SP

HOST/SP

RestrictedKrbHost/SP.contoso.com

HOST/SP.contoso.com

Existing SPN found!

PS C:\Users\Administrator.contoso> setspn -X

Checking domain DC=contoso,DC=com

Processing entry 0

found 0 group of duplicate SPNs.

So, for same Service there should not be 2 different Service accounts for which SPNs are registered otherwise IIS will not know which account to decrypt from.

Tools:

Klist
Kerberos Authentication tester
Fiddler
Microsoft network monitor

Klist.EXE:

this command-line tool helps you list and purge tickets on your client machine. You can list tickets to verify the validity of tickets. You can also check for other details, such as the permissible flags, SPN type, the client, and server details.

KLIST output (in this example, for a Site URL: https://kerberos:9090):

PS C:\Users\Administrator.contoso> klist

Current LogonId is 0:0x16958c

Cached Tickets: (2)

#0> Client: Administrator @ CONTOSO.COM

Server: krbtgt/CONTOSO.COM @ CONTOSO.COM

KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96

Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize

Start Time: 5/6/2018 21:43:29 (local)

End Time: 5/7/2018 7:43:29 (local)

Renew Time: 5/13/2018 21:43:29 (local)

Session Key Type: AES-256-CTS-HMAC-SHA1-96

Cache Flags: 0x1 -> PRIMARY

Kdc Called: DC.contoso.com

#1> Client: Administrator @ CONTOSO.COM

Server: HTTP/kerberos @ CONTOSO.COM

KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)

Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize

Start Time: 5/6/2018 21:43:29 (local)

End Time: 5/7/2018 7:43:29 (local)

Renew Time: 5/13/2018 21:43:29 (local)

Session Key Type: RSADSI RC4-HMAC(NT)

Cache Flags: 0

Kdc Called: DC.contoso.com

So, if server does not have same Decryption mechanism (Say RC4) so it can't decrypt tickets. In that case, make sure you enable AES encryption for Service account properties.

Run these two commands below if you want to Purge user and System level Kerberos Tickets.

PS C:\Users\Administrator.contoso> klist purge

Current LogonId is 0:0x16958c

Deleting all tickets:

Ticket(s) purged!

PS C:\Users\Administrator.contoso> klist -li 0x3e7 purge

Current LogonId is 0:0x16958c

Targeted LogonId is 0:0x3e7

Deleting all tickets:

Ticket(s) purged!

Kerberos Authentication Tester (It can be found here: https://blog.michelbarneveld.nl/michel/archive/2009/12/05/kerberos-authentication-tester.aspx)

Kerberos Authentication Tester Features:

It shows what authentication method is used in a web request: None, Basic, NTLM or Kerberos
It shows the SPN used in case of Kerberos
It shows the HTTP status
It shows the HTTP Headers of the request.
It shows the version of NTLM used (v1 or v2)
It has a detailed view with a complete breakdown of the Authorization header. (Yep, went through all the RFCs to dissect the Kerberos and NTLM packages)
It shows your current Kerberos tickets and allows you to remove them (like klist.exe)

The blue details link at the top right shows a new screen with the details of the request, including a breakdown of the Authorization HTTP Header. You can also save this XML.

It shows all kerb Tickets.

Network Monitor/WireShark: network trace capture and review tools. You can review trace to identify Kerberos errors.

Best way to get the network capture is:

1. To reduce the possibility of caching data, do one of the following:

•Close/Reopen client application

•Logoff/Logon client workstation

•Reboot client workstation

2. Start the network capture

3. Clear DNS cache using:

ipconfig /flushdns

4. Clear NetBIOS cache using:

nbtstat –RR

5. Clear user Kerberos tickets using:

klist purge

6. Clear system / computer Kerberos tickets using (Vista or higher only):

Klist –li 0x3e7 purge

7. Reproduce the authentication failure with the application in question

8. Stop the network capture

Now that you have the capture, you can filter the traffic using the string 'Kerberosv5' if you are using Network Monitor. If you are using Wireshark, you can filter using the string 'Kerberos'.

To be more thorough, load the Authentication Traffic filter that shows packets containing Kerberos tickets as well. You can do this by clicking the Load Filter button, choose Standard Filters, and then click Authentication Traffic.

OR add this Filter:

KerberosV5

OR KerberosV5.KrbError.Sname

OR KerberosV5.AsReq

OR KerberosV5.AsRep

OR KerberosV5.TgsReq

OR KerberosV5.TgsRep

Click 'Apply' to begin filtering…

In my Network trace examples below:

I am using SQL as my Client machine

192.168.2.50 is DC (KDC)

SP is SharePoint server

Analysing Network trace……………………….

First, we see AS (Authentication Server) request and response.

DC given AS response Ticket (TGT) to client.

(Service name) Sname requested and sent is krbtgt:

Sname: krbtgt/CONTOSO.COM

Then Client uses AS response Ticket to make TGS request (Service ticket) to KDC.

TGS Response ticket (Service Ticket) is sent to client by KDC.

(Service name) Sname requested and sent is HTTP:

Sname: HTTP/WebApp name

FIDDLER:

1st request will be anonymous if user is not authenticated.

401 is what we see first ……………..

Then we see 302 client Presenting kerberos Ticket to Server (SP) where server responds with a Kerberos Reply after decrypting the ticket with App Pool Identity.

If the Ticket presented by Client is valid then server moves request (302) to the page Client should be going to..

It could be home page of the Site, as in this example is:

Location: https://teams/_layouts/15/start.aspx#/SitePages/Home.aspx