Kerberos and Load Balancing

Hi guys, Joji Oshima here again. Today I want to talk about configuring Kerberos authentication to work in a load-balanced environment. This is a more advanced topic that requires a basic understanding of how Kerberos works. If you want an overview of Kerberos, I would suggest Rob’s excellent post, Kerberos for the Busy Admin. In this post, I will be using a load balanced IIS web farm as the example, but the principal applies for other applications.

The Basics:

As you may know, Kerberos relies on Service Principal Names (SPNs). SPNs are associated with objects in Active Directory and registered under the servicePrincipalName attribute.

If you are using 2008 Server or higher, you can view this attribute using Active Directory Users and Computers (ADUC) with Advanced Features enabled and going to the Attribute Editor tab. Click the View menu and then select Advanced Features

You can also view attributes using ADSI Edit (adsiedit.msc) or the Setspn command line tool.

When an application makes a request for a Kerberos ticket, it makes that request for a specific SPN (like http/server01.contoso.com). The Key Distribution Center (KDC) will search Active Directory for the object that has that principal name registered to it, and encrypt the ticket with that object’s password. The object that is running the service has the same password, so when the ticket arrives, it can decrypt it.

The Problem:

If you have a single IIS server, the service is typically running under Local System. The standard SPNs are registered to the computer account (like host/server01 & host/server01.contoso.com)* so when a request for http/server01 comes in, the ticket will be encrypted using the computer account’s password. This configuration works well for a single server environment.

*The host SPN works for many services including http. If there is a specific entry for http, it will use that, otherwise it will fallback and use host.

In a load-balanced environment, users will access the service using a unified name instead of the individual servers. Therefore, instead of accessing server01.contoso.com or server02.contoso.com, they will access myapplication.contoso.com. In this scenario, there are two computer accounts, so where do you register the Service Principal Name? One idea would be to register the principal name to both computer accounts. The problem with this idea is that Service Principal Names must be unique. When the request comes in for http/myapplication.contoso.com, the KDC would not know which object’s password to encrypt the ticket with, so it will return an error. You will also see Event ID 11 populating your event logs if you have any duplicate SPNs in your directory.

The Solution:

Instead of running the service under Local System, have each server run the application using a specific service account. In IIS, you can accomplish this by having the application pool run under the service account. Here is how you would set it in IIS7. You could also follow the instructions in this TechNet article.

1. Open Internet Information Services (IIS Manager) and browse to the Application Pools page

2. Right click the application pool that is running your site, and choose Advanced Settings

3. Under Process Model, look for Identity and click the button to the right

4. This will bring up a dialog box where you can choose what credentials to use for the application pool. Choose Custom account and click the set button.

5. Enter the credentials for your service account and click ok

6. If you are using IIS7 and have Kernel Mode Authentication set, you will need to do one additional step. Open the ApplicationHost.config file and enable the useAppPoolCredentials setting. IIS7 added the option to authenticate users in Kernel mode to speed up the authentication process. By default, it will use the computer account for authentication requests even if the application pool is set to a service account. By changing this setting, you get the benefits of Kernel Mode Authentication, but still authenticate with the service account.

Sample ApplicationHost.config change:

<system.webServer>
<security>
<authentication>
<windowsAuthentication enabled="true" useAppPoolCredentials="true" />
</authentication>
</security>
</system.webServer>

After you have the services running under the same service account, register the unified name, http/myapplication.contoso.com, to that service account. No matter which server the client is routed to, the service will be able to decrypt the ticket using its password. You can register SPNs using the command line tool Setspn.

Extra Credit:

Suppose you want the ability to use Kerberos authentication accessing the servers individually and using the unified name. Currently, if you request a ticket for http/server01.contoso.com, the KDC will encrypt the ticket using server01’s computer object password. The service is not running under local system, so it will not be able to decrypt that ticket. However, you can register additional SPNs to the service account. In this scenario, you could register the following SPNs to the service account.

You can also view the attributes currently registered to an account using the Setspn command line tool.
Syntax: Setspn –l accountname

This will not interfere with the host SPNs registered to the computer account. When an incoming request comes for http/server01, it will check for the exact string first. If it cannot find it, it will look for host/server01.

Problems:

Remember that a Service Principal Name can only be registered on one account at a time. If you are using 2008 Server or higher, you can search for duplicate SPNs in your environment by using the command: Setxpn -f -q http/myapplication*

You can also use the command line tool LDIFDE to find duplicate SPNs. The command below will output a text file called SPN.txt that contains all objects with a service principal name that starts with http/myapplication. This file will be located in the same directory you run the command in unless you specify a path in the –f switch.

LDIFDE -f spn.txt -r (serviceprincipalname=http/myapplication*) -l serviceprincipalname

• The –f switch determines the output file
• The –r switch determines the search criteria, and the * at the end is a wildcard
• The –l switch chooses which attributes will be listed in the output file

Final Thoughts:

There are many benefits to using Kerberos authentication but configuring it properly may feel like a daunting task, especially in a more complex environment. I hope this post makes configuring this a bit easier.

- Joji “adult swim in the app pool” Oshima

Comments

• Anonymous
  August 09, 2011
  > Suppose you want the ability to use Kerberos authentication accessing the servers individually and using the unified name. do you guys see lot of customers wanting exactly this? Weird.

• Anonymous
  August 09, 2011
  If we had a big button that said "push this to delete all your data", a certain percentage of our 2 billion customers would do it. :) Our customer base is just incomprehensibly gigantic.

• Anonymous
  August 09, 2011
  Ever present note about ADUC/Attribute editor - you must browse to the object because you can't use find. I want a big delete button...

• Anonymous
  August 09, 2011
  The comment has been removed

• Anonymous
  August 09, 2011
  That's how I solve cases: I just hit the $#&%^*ing awesome button. You didn't seriously think I had any idea what I was talking about here, did you?

• Anonymous
  August 09, 2011
  Shhhh. Don't cannibalize my work. We need to sell consulting services as well. In fact, the only product that doesn't have that button built in is PowerShell. They have to use a cmdlet for that. That's why we prefer Opalis (it's much more expensive also).

• Anonymous
  August 09, 2011
  New buttons in the next releases? Very excited now :)

• Anonymous
  August 10, 2011
  Okay, me haz to say something seriouz. It's worth to mention that absolutely the same concepts apply not only to Load Balansing but to Failover Clustering as well. If you intend to end up with using Kerberos with a clustered service, you should not run it using Local System, Local Service or Network Service. Instead, you'd need to create a domain account for that and assign a service-specific SPN to it (e.g. MSSQLSvc/ClusteredName.contoso.com).

• Anonymous
  August 10, 2011
  Great article: Find duplicate SPN's, Windows 2008 & up. Why not use setspn -X :-)

• Anonymous
  August 10, 2011
  @Silvester99 Good call. That will find all duplicates throughout the domain/forest though, not just the one you care about right now. It can also be incredibly slow - imagine running that in Microsoft's internal forest, with millions of principals to check. It's more of an overall inventory tool. If you want to make sure it runs accurately too, you need to use: setspn –x –f @Pronichkin Your cluster point is excellent and well-taken

• Anonymous
  August 10, 2011
  Remeber in Powershell.. you can create a GUI for the "Make everyting @#$%ing awesome" button & make all of your scripts awesome-er :) Seriously now.. a few questions:

1. Is there a limit to the # of SPNs an service account can have?
2. Why did I ask #1? In our case we're deploying a large Exchange 2010 rollout over 5-6 (mabye more) datacentres using F5's products for load balancing. In each site we using LTM to load balance SITEmail.domain.com (SITE being an internal DC code like TOR)BUT we're trying to use GTM for global load balancing to create a global CAS namespace. Why? we've integrated OWA into our portal (integrated inbox on home page as a webmail client) and its much easier to tell our web developers to just point all clients to a single namespace (e.g. mail.domain.com) and let F5 route them to the most "geographically convenient" cas array (yup, were using cas 2 cas proxy internally and in testing things work wonderfully and are supported by MS) My concern with #2 is that if we go this route it looks like we'll need to setup a single global sevice (to maintain a uniquite mail.domain.com HTTP SPN)account and use the MS recommended RollAlternateServiceAccountPassword.ps1 script to manage said sevice account. See blogs.technet.com/.../recommendation-enabling-kerberos-authentication-for-mapi-clients.aspx for info on exchange. In the above case I need to create one spn for the global namespace mail.domain.com AND spns for each LTM VIP so after a while I'll have a whole wack of records (4 per DC site) Soo... anyone see any issues with a large volume of SPNs? Or for the exchange minded a single service account for ASA? Feedback would be appreciated!

• Anonymous
  August 10, 2011
  The comment has been removed

• Anonymous
  August 11, 2011
  >> Suppose you want the ability to use Kerberos authentication accessing the servers individually and using the unified name. >do you guys see lot of customers wanting exactly this? Weird. I wouldn't say a lot, but it's been asked before. If I recall correctly, they used network captures for troubleshooting and wanted know which web server to start the trace on (without reconfiguring the load balancer).

• Anonymous
  August 12, 2011
  >>If you have a single IIS server, the service is typically running under Local System Poor, poor customers... Using local system - the most powerful account - to run the darn web services? That's what is weird. Probably we should try Enterpise Admin for that? =)))