Share via

All About Kerberos “The Three Headed Dog” with respect to IIS and Sql

  • Article

Kerberos Understanding and troubleshooting

Steps to be followed while dealing with Troubleshooting Kerberos related issues.

Prerequisites:

1) Kerberos can only work in intranet because client needs to interact with the Domain controller directly.

2) Kerberos can be used in internet only if you have a public facing active directory. But public facing active directory is not recommended due to security concerns.

3) Server , client and dc must be in the intranet.

Tools that come handy:

SETSPN (For adding, listing, deleting SPN entries for a domain)

KERBTRAY (on client machine)- For checking the Kerberos ticket used by the client to access a web server. It gives you information as to which ticket is being used by the client to access the IIS server, and whether the ticket is capable of delegation.

Deleg Config(server side tool) that adds a application under the side you are trying to browse which gives you information if the website is working for single hop and double hop

Klist(client side) to check which all tickets I am using.

Netmon

How Kerberos Works:

Overview in general:

Single domain Scenario

clip_image001

1) Client will contact kdc saying that it’s a user and it needs a ticket to get tickets (TGT)

2) KDC will give the TGT to the client by encrypting with the client machine password hash.

3) When client wants a service ticket, it will produce its TGT(for authorisation) and request TGS.

4) Based on auth the service ticket is given to the client.

5) Then the client passes its service ticket to the network server or resource machine for authentication.

6) After authentication, normal client server response request process continues.

Multiple Domain Scenario

clip_image002

Both Child1 and Child2 are in the same forest with the same parent domain R2dom.local.

Administrator of the Child domain (CHILD1) login to a member server (CH1-Mem) in CHILD1 domain.

After login in the user tries to access \\r2dom-ch2-Mem1 . R2dom-ch2-Mem1 is a member server in Child2 domain.

--> I have used Network monitor to analyze and understand how Kerberos authentication would work, when accessing resource across domain.

Below you see that the Administrator is getting the required Kerberos tickets when accessing resources across domain.

10.2.1.2   = DC of CHILD1.R2DOM.LOCAL10.1.1.2   = DC of R2DOM.LOCAL10.3.1.1   = Dc of CHILD2.R2DOM.LOCAL10.10.10.1   = CH1-Mem in CHILD110.10.10.1   10.2.1.2   KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1                                                                           --> KrbApReq: KRB_AP_REQ (14)                                                             Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/CHILD1.R2DOM.LOCAL

 

** Here you see a Kerberos TGS request being sent to the local domain (CHILD1.R2DOM.LOCAL) DCs for a SPN cifs/r2dom-ch2-Mem1. Local domain TGT sent in the TGS request.

10.2.1.2   10.10.10.1   KerberosV5:TGS Response Cname: Administrator                                              --> Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/R2DOM.LOCAL

** Here the local domain DC returns a TGT of the Parent domain R2DOM.LOCAL. This is like a referral being sent to the client as the local domain does not have the right to issue a Kerberos Ticket for cifs/r2dom-ch2-Mem1.

10.10.10.1   10.1.1.2   KerberosV5:TGS Request Realm: R2DOM.LOCAL Sname: krbtgt/CHILD2.R2DOM.LOCAL --> KrbApReq: KRB_AP_REQ (14)    Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/R2DOM.LOCAL

** Now the client machine send a TGS request to the Parent domain R2DOM.LOCAL, requesting for a TGT of another of its Child domain where the cifs/r2dom-ch2-Mem1 resides. When sending this request to the parent domain, the client uses the TGT of the Parent domain received in the earlier referral from local DC.

10.1.1.2   10.10.10.1   KerberosV5:TGS Response Cname: Administrator                                              --> Ticket: Realm: R2DOM.LOCAL, Sname: krbtgt/CHILD2.R2DOM.LOCAL

**Parent domain sends the TGT of the child2 domain to the client. This can also be taken as a referral to the CHILD2 domain.

10.10.10.1   10.3.1.1   KerberosV5:TGS Request Realm: CHILD2.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1                                              -->KrbApReq: KRB_AP_REQ (14)                                                             Ticket: Realm: R2DOM.LOCAL, Sname: krbtgt/CHILD2.R2DOM.LOCAL

** Eventually the Child sends a TGS request for the SPN cifs/r2dom-ch2-Mem1 to the DCs in domain CHILD2.R2DOM.LOCAL who is authorized to issue a ticket for the server (r2dom-ch2-Mem1) in its domain. This time the client uses the CHILD2 domains TGT to make request.

10.3.1.1   10.10.10.1   KerberosV5:TGS Response Cname: Administrator                                              --> Ticket: Realm: CHILD2.R2DOM.LOCAL, Sname: cifs/r2dom-ch2-Mem1

** Finally the client gets the Kerberos Ticket for  cifs/r2dom-ch2-Mem1, which will help the Administrator user access the shares on r2dom-ch2-Mem1 in domain CHILD2.R2DOM.LOCAL.

The workstation the user 'Administrator" is using to access the resource across domain, also needs a similar ticket.

Tickets for r2dom-ch2-Mem1 is requested from local domain, who returns the error KDC_ERR_S_PRINCIPAL_UNKNOWN, as the r2dom-ch2-Mem1 computer is not a part of the local domain (CHILD1.R2DOM.LOCAL)

10.10.10.1   10.2.1.2   KerberosV5  KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: r2d-ch2-Mem1 10.2.1.2   10.10.10.1   KerberosV5  KerberosV5:KRB_ERROR  - KDC_ERR_S_PRINCIPAL_UNKNOWN (7)

Below you see that the client computer is getting the required Kerberos tickets when accessing resources across domain.

10.10.10.1   10.2.1.2   KerberosV5  KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1 10.2.1.2   10.10.10.1   KerberosV5  KerberosV5:TGS Response Cname: CH1-Mem$ 10.10.10.1   10.1.1.2   KerberosV5  KerberosV5:TGS Request Realm: R2DOM.LOCAL Sname: krbtgt/CHILD2.R2DOM.LOCAL 10.1.1.2   10.10.10.1   KerberosV5  KerberosV5:TGS Response Cname: CH1-Mem$ 10.10.10.1   10.3.1.1   KerberosV5  KerberosV5:TGS Request Realm: CHILD2.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1 10.3.1.1   10.10.10.1   KerberosV5  KerberosV5:TGS Response Cname: CH1-Mem$

Kerberos from IIS perspective , how it works:

1) Client sends a HELLO to the server

2) Server replies with a 401(sub status 2) Unauthorized telling the client that it requires to authenticate itself before accessing the page using www authenticate header (negotiate provider).

3) Client requests the Domain controller saying I am trying to access a particular website(say bing.com by specifying it as http/bing.com) from a particular server.

4) Domain controller looks through its SPN (service principal name) table for the particular application or site which will have the entries as below.

Service| Name of the service registered| registrar name (account name) Eg: HTTP| bing.com| Chiranth

This means that bing.com is registered for the account Chiranth

Once it finds an entry in the table then DC will encrypt the token with the password hash of the account and send it to the client

5) The Client now sends the token to the server.

6) The server uses the account’s password hash to which the spn is registered(will use machine account if you have kernel mode authentication checked or will use application pool credentials if you have useapppoolcredentials as true in IIS (system.webServer/security/authentication/windowsAuthentication) to decrypt the token. If the server is able to successfully decrypt the token the requested page is given back.

NOTE : To register SPN use the command

setspn –a http/[fqdn server name] domain\account name

Specific Scenario when cx wants to register the host or machine spn to domain account, follow the below steps:

setspn –a http/<computer-name> <domain user account>setspn –a http/<FQDN> <domain user account>

1) When you use kernel mode authentication we use Host SPN’s password for encryption and decryption of the token  by DC and server

2) So if you register an SPN for the account name check to useapppoolcredentials in the configuration editor of windows authentication.

Why Kerberos and not NTLM:

· Kerberos authentication offers the following advantages over NTLM authentication:

· Mutual authentication. When a client uses the Kerberos v5 protocol for authentication with a particular service on a particular server, Kerberos provides the client with an assurance that the service is not being impersonated by malicious code on the network.

· Delegation support. Servers that use Kerberos authentication to authenticate clients can impersonate those clients and use the client's security context to access network resources.

· Performance. Kerberos authentication offers improved performance over NTLM authentication.

· Simplified trust management. Networks with multiple domains no longer require a complex set of explicit, point-to-point trust relationships.

· Interoperability. Microsoft's implementation of the Kerberos protocol is based on standards-track specifications recommended to the Internet Engineering Task Force (IETF). As a result, the implementation of the protocol in Windows 2000 lays a foundation for interoperability with other networks where Kerberos version 5 is used for authentication.

Basic Troubleshooting steps:

What should we do when we see that we are getting 401 unauthorized or windows authentication failure.

Client Side

1) Check if Windows Integrated Authentication is enabled on the browser. Automatic logon with current username and password or, Automatic logon only in Intranet Zone.

2) Check if the URL is on Intranet Zone Sites.

Server side

3) Check if Basic Authentication is working. If it is not working, then it is not Kerberos issue.

4) If basic works, check for NTLM and if it doesn’t work, we should involve AD.

5) In a successful scenario when Kerberos works you will see a 401.2 and 200

clip_image001

6) If it is going over NTLM we will see 401.2 401.1 and 200 in fiddler.

clip_image001[7]

7) Also to check if the token is Kerberos or NTLM, the Kerberos ticket usually starts with Y and usually NTLM token starts with TLRM.

Kerberos Ticket eg: Authorization: Negotiate YIIQiwYGKwYBBQUCoIIQfzCCEHugMDAuBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICHgYKKwYBBAGCNwICCqKCEEUEghBBYIIQPQYJKoZIhvcSAQICAQBughAsMIIQKKADAgEFoQMCAQ6iBwMFACAAAACjgg6sYYIOqDCCDqSgAwIBBaEcGxpGQVJFQVNULkNPUlAuTUlDUk9TT0ZULkNPTaIeMBygAwIBAqEVMBMbBEhUVFAbC2NoaXJhbnRoOHIyo4IOXTCCDlmgAwIBEqEDAgEQooIOSwSCDkewzlFPUeEgCJxtUmS7i1lNBiKYS9OCRPEHdyUytqKqs9SY
NTLM Token eg: Authorization: NTLM TlRMTVNTUAADAAAAGAAYAIIAAACUAZQBmgAAAA4ADgBYAAAADAAMAGYAAAAQABAAcgAAAAAAAAAuAgAABYKIogYBsR0AAAAPnYnxQFaTOYGkUPDVgZ7SckYAQQBSAEUAQQBTAFQAYwBoAGkAcgBhAG0AVAAtAEMASABJAFIAQQBNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAN6Op+WbvZB0Tchmb777NmIBAQAAAAAAAB5e5pMbts4B1nm7EWkuiDMAAAAAAgAOAEYAQQBSAEUAQQBTAFQAAQAWAEMASABJAFIAQQBOAFQASAA4AFIAMgAEADQAZgBhAHIAZQBhAHMAdAAuAGMAbwByAHAALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAAMATABjAGgAaQByAGEAbgB0AGgAOAByADIALgBmAGEAcgBlAGEAcwB0AC4AYwBvAHIAcAAuAG0AaQBjAHIAbwBzAG8AZgB0AC4AYwBvAG0ABQAkAGMAbwByAHAALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAAcACAAeXuaTG7bOAQYABAACAAAACAAwADAAAAAAAAAAAQAAAAAgAACXZ723Po2/mq+xx6VA97FdnWz3JNKjqzWGOYp01eYk9QoAEAAAAAAAAAAAAAAAAAAAAAAACQAgAEgAVABUAFAALwBjAGgAaQByAGEAbgB0AGgAOAByADIAAAAAAAAAAAAAAAAA

8) To verify this we can copy this token value and decode it from base64 in fiddler tools option.

9) Create a test page and test with windows authentication(single hop).

10) If you see that the token sent to the server seems to be the Kerberos token and still it is not working. the First thing to check would be if you are using the right account fro decryption of the ticket n the server. the most awesome thing is if you are not using the proper account fro decryption you will get notified about this by the Kerberos system event log entries on the client itself if Kerberos

Type : ErrorDate : Time : Event : 4Source : KerberosCategory : NoneUser : N/AComputer : xxxxxxDescription: The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server . The target name used was HOST/xxxxx or HTTP/xxxxx. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN is only registered on the account used by the server. This error can also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server and the KDC are both configured to use the same password. If the server name is not fully qualified, and the target domain () is different from the client domain (xxx.xxx.xxx), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

11) SPN configuration

a. Check if Kernel mode Authentication is enabled, if yes, then spn must be registered for machine account

b. If Kernel mode is not enabled and app pool identity is domain account then SPN must be registered for the domain account

c. If Kernel mode authentication is enabled and useapppoolcredential is set to true(as described above) and app pool credential is domain account, then the SPN must be registered for domain account.

12) To check if u have duplicate spn’s u can try the commands.

To find duplicate spn’s on server

Use command prompt to execute this command

Setspn –x

Or you could run the below in DC

To find duplicate SPNs
Use command prompt to execute this command on dc:

ldifde -f <filename> -d "<dc=domain-netbiosname,dc=primary-domain>" -l serviceprincipalname -r "(serviceprincipalname=<serviceprincipalname-to-check-for-duplicates>)" -p subtreee.g. if the domain name is test.abcd.com:ldifde –f C:\log.txt -d "dc=test, dc=abcd, dc=com"-l serviceprincipalname –r "(serviceprinicpalname=http/test.abcd.com)" -p subtree

On the server side u can run

Setspn –l  acctname

Also you can run the below command to list the spn’s in the entire FOREST for the particular Hostname.

Setspn –F –Q service/hostnameEg: Setspn –F –Q */chiranth8r2.com

This will list all the service spn’s for chiranth8r2.com and also will tell you to which account it is registered.

Note : If everything looks fine and still single hop is failing. check which account is application pool running under. if it is running under application pool identity, change it to local system and give it a try. if test with local system succeeds, then there is a hotfix from Microsoft for it to work under applicationpoolidentity. Below is the hotfix link. https://support.microsoft.com/kb/2545850

13) Create a virtual directory under the site pointing it to a shared location with a simple page(double hop delegation test). If the test fails follow step 13 or if it succeeds follow step 14.

14) (For double Hop) If your test page works check if the trusted for delegation setting is checked in active directory for the account to which machine is registered. If you are using kernel mode you have to trust the machine or host for delegation. To check if machine account is trusted for delegation got ‘Active directory users and computers’, right click on the machine name on go to the tab delegation and select the second option ‘Trust this computer for delegation to any service’ or the third option. If you are using app pool credentials trust the app pool account for delegation. Same above steps should be followed for app pool account in AD.

15) If trusted for delegation is already present and test with shared location succeeds, then there must be some issues with the SQL spn’s. Below are the steps we can follow.

16) How to Verify if SQL is good to except kerberos

1.First thing you would like to check is if your SQL is already capable to Kerberos or not. Simple way to verify this is to see if there is any existing connection using Kerberos. You could do this by running following command against your SQL instance and reviewing the result. If, even a single connection says Kerberos, You are good and don’t need to proceed further.

select auth_scheme from sys.dm_exec_connections

There could be combination of these protocols you would typically see as a result of above query.

NTLM, Kerberos, SQL

2. If the list does not show Kerberos, We would proceed further.  Now, we have to check if Service Principal Names (SPN) for your SQL instance are configured properly or not.

In command below, I am presuming that your SQL box is called MySQLServer.

You domain is called MyDomain.

SQL Service account is SQLServiceAccount

Check the Service account of your SQL Service and run following command against setspn .

SetSPN -L MyDomain\SQLServiceAccount

If this is a standalone default SQL instance, It should return following two Service Principal Names (SPN).

MSSQLSvc/MySQLServer.MyDomain.com:1433 MSSQLSvc/MySQLServer:1433

If this is a named instance  listening on TCP port 4545, SPNs should look like this.

MSSQLSvc/MySQLServer.MyDomain.com:4545 MSSQLSvc/MySQLServer:4545

If this is a Clustered default SQL instance, It should return following two Service Principal Names (SPN).

MSSQLSvc/MySQLVirtualName.MyDomain.com:1433 MSSQLSvc/MySQLVirtualName.MyDomain.com

If this is a Clustered named SQL instance  listening on TCP port 4545, SPNs should look like this

MSSQLSvc/MySQLVirtualName.MyDomain.com:4545 MSSQLSvc/MySQLVirtualName.MyDomain.com 

3.If you see any of the required set of the SPN missing, You need to create it explicitly. For example you run following command.

SetSPN -L MyDomain\SQLServiceAccount

And it returned only one SPN .

MSSQLSvc/MySQLServer:1433

You have to create another required one using following command .

SetSPN -A MSSQLSvc/MySQLServer.MyDomain.com:1433 MyDomain\SQLServiceAccount

Please note, adding SPNs requires domain admin privileges. Once added successfully, You could confirm it using following command. SetSPN -L MyDomain\SQLServiceAccount it should return .

MSSQLSvc/MySQLServer.MyDomain.com:1433 MSSQLSvc/MySQLServer:1433

4.Once you have set of SPNs in place, your SQL Server is good to accept Kerberos authentication. You have to close your existing applications or connections and open them again to initiate fresh Kerberos connections.

5.In some cases, You would see your connections still using NTLM (see point 1). This would happen most probably because of similar SPNs existing under some other Service account, These would be considered as duplicate SPNs. This can happen if you have your SQL Service account being changed recently. As a result, There would be old SPNs still present under old account. We can find duplicate SPNs by running following command using SetSPN.

SetSPN -X

In case there is same SPN registred under some other account, say OLDSQLServiceAccount, You would see it in output of SetSPN -X. You can then get rid of this duplicate SPN using following command.

SetSPN -D MSSQLSvc/MySQLServer.MyDomain.com:1433 MyDomain\OLDSQLServiceAccount

16) The dns entry for the url should be marked as an A record or else it will give u intermittent prompts for username and password(two types a record and c record).

17) Disable LoopBack Check if you are facing the problem only on the Server Side. (but able to browse the url from clients, as on server locally it works over ntlm)

18) You can also enable Kerberos event logging. On information about how to enable Kerberos event logging. see below or the kb https://support.microsoft.com/kb/262177

Start Registry Editor. Add the following registry value: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\ParametersRegistry Value: LogLevelValue Type: REG_DWORDValue Data: 0x1If the Parameterssubkey does not exist, create it.

Note Remove this registry value when it is no longer needed so that performance is not degraded on the computer. Also, you can remove this registry value to disable Kerberos event logging on a specific computer.

Quit Registry Editor. The setting will become effective immediately on Windows Server 2008, on Windows Vista, on Windows Server 2003, and on Windows XP. For Windows 2000, you must restart the computer.

You can find any Kerberos-related events in the system log.

19) You can check in deep by downloading delegconfig tool and put that application inside the website you are trying to browse.

20) If none of the above steps help, we can collect a netmon trace on client, server and AD simultaneously and see the communication. We can see if we are getting a service ticket(TGS) and also isolate if the issue lies in AD side, client side or server side.

Note: One of my colleagues has recently developed a tool to troubleshoot the Kerberos issue and manage the settings and information regarding that can be found in the below link

https://blogs.msdn.microsoft.com/surajdixit/2018/02/07/kerberos-configuration-manager-for-internet-information-services-server/

https://github.com/SurajDixit/KerberosConfigMgrIIS

SPN BLOGS and Kerberos Troubleshooting Blogs: https://blogs.iis.net/brian-murphy-booth/archive/2007/03/09/the-biggest-mistake-serviceprincipalname-s.aspx https://blogs.msdn.com/b/saurabh_singh/archive/2007/01/29/kerberos-troubleshooting-from-iis-perspective.aspx https://blogs.msdn.com/b/saurabh_singh/archive/2008/11/16/who-knows-you-may-be-missing-these-points-for-kerberos-authentication-failures-for-web-applications.aspx

Technorati Tags: authentication,iis,UI,missing section,missing,anonymous,registry,kerberos,kerberos troubeshooting,spn,fiddler,http spn,sql spn,windows authentication,negotiate,usekernelmode

Comments

  • Anonymous
    October 15, 2013
    Chiranth, excellent post! Keep up the good work.Jeff

  • Anonymous
    October 18, 2013
    thanks Jeff :)

  • Anonymous
    April 02, 2014
    Awesome Blog Chiru :)

  • Anonymous
    August 14, 2014
    Very useful blog Chiranth :)

  • Anonymous
    September 09, 2015
    Nice post

  • Anonymous
    September 23, 2015
    The blog is very helpful which provides clear understanding of the concept!!

  • Anonymous
    February 19, 2016
    Nice article Chiranth.  I think  one minor correction needs to be made.  Near the top of the article, your step number 5 as written is: "Then the client passes its TGS to the network server or resource machine for authentication." My comment is that the TGS is a process which runs only on the KDC.  Therefore step 5 should be modified to: "Then the client passes its service ticket to the network server or resource machine for authentication." -- Best Regards, Todd Heron | Active Directory Consultant

  • Anonymous
    February 21, 2016
    @Todd: Thanks for pointing it out. i have modified it accordingly :)

  • Anonymous
    September 05, 2016
    Hi Chiru,I am new to learn kerberos,Could you please guide me, how can we configure or implement kerberos authentication in .net application.Please reply with details explanation which will be useful for me.Thanks in advance.Ali

    • Anonymous
      September 22, 2016
      Sorry for the late response. i was not monitoring the comments as i was away. You can follow the below blog to setup windows auth or kerberos for a IIS sitehttps://blogs.msdn.microsoft.com/chiranth/2014/04/17/setting-up-kerberos-authentication-for-a-website-in-iis/

  • Anonymous
    April 27, 2017
    after following the configurations for machine account Kerberos delegation inhttps://blogs.msdn.microsoft.com/chiranth/2014/04/17/setting-up-kerberos-authentication-for-a-website-in-iis/I am facing weird issue. that I can browse to the second hup from my client machine only after first login from IIS server specially after application pool recycle.

    • Anonymous
      May 07, 2017
      I would suggest that you enable failure log in auditing on both iis server and backend machines and see if the event logs during that time.Also if your machine is IIS 7.5 check if you have the following fix appliedhttp://support.microsoft.com/kb/2545850

  • Anonymous
    June 05, 2017
    Thanks Chiranth. It was very helpful.

  • Anonymous
    August 18, 2017
    Very nice article and its clarified my queries regarding the fiddler logs.