Using SCOM to Detect Golden Tickets

Disclaimer: Due to changes in the MSFT corporate blogging policy, I’m moving all of my content to the following location. Please reference all future content from that location. Thanks.

For the three people that religiously read my blog, you know by now that I’ve been writing quite a bit on using SCOM to detect some of the anomalous events that are specific to an intruder in your environment.  This is the first, of what will hopefully be a few rules needed to detect the existence of the back doors that attackers leave behind so that they can re-enter your environment at will.

As a quick matter of definition, let’s boil this down to the basics.

• In order to obtain a golden ticket, your environment had to be already compromised:  An attacker gets them by backing up the master Kerberos ticket’s key.  In a windows environment, this key only exists on a domain controller, which means that if the attacker acquired it, they had already compromised your DC.
• Once the attacker has obtained this key, they can re-enter your environment at will, even if you’ve already evicted them: This is the big danger to golden tickets in that they don’t need to steal credentials of a domain admin again in order to re-enter the environment. They simply create themselves a ticket under any user context and give it domain admin rights. They can re-enter the environment and steal your AD database all over again or grab whatever else that they intended to come back and get.  It’s a forged krgtgt ticket, and as such, they don’t even need an account in your environment.
• They have much longer expiration periods than a standard ticket:  The default expiration of a ticket is 10 hours in active directory. This is configurable, so it may be different in every environment, but in order for it to work, the attacker wants to have a long period of time in which they can re-enter an environment. When mimikatz, for instance, is used to generate a golden ticket, the default expiration of this ticket is 10 years.
• They rely on having a valid Kerberos TGT key: This is the kicker to protecting yourself from them, but as long as the key used to sign any forged ticket is valid, the attacker can still re-enter your environment.
• The krgtgt account’s password does not change automatically: The password itself is known only to DCs, and if you manually change it, the password you specify won’t be the actual password to the key, but by default this password does not reset. If you’re at a 2003 domain functional level and go to a 2008 domain functional level, this will do a one time reset of the password as a part of that process. Beyond that, however, this password does not change. Truthfully, this should be changed periodically, but most organizations do not do it. I’d note that the new AD MP for SCOM (if you configure client monitoring) will generate alerts on the krbtgt password if it has not been reset recently.  Microsoft does have instructions online, and it has PowerShell scripts which make it easier.
• Standard eviction can often leave you exposed to this back door:  Simply removing the attacker’s malware does not protect you.  This will need to be done in conjunction with an AD recovery. You will need to both recover the AD environment (i.e. change all the passwords), remove the malware, and reset the krbtgt password… twice.  The reason for this is that Active Directory holds a copy of two keys:  the current krbtgt key as well as the previous key.  Both the current and previous keys need to be scrubbed from memory to invalidate any existing golden tickets.
• This is not a vulnerability unique to windows:  The purpose of this blog is monitoring the Windows environment, but any Kerberos environment is vulnerable to this type of an attack.  It is the price one pays if their tier 0 is compromised.
• Following the procedures below will not prevent an attacker from entering the environment:  All this will do is break existing golden tickets. If the attacker has not been completely evicted from your environment, they can and will get in.  They can then turn around and regenerate a new one.  There is definitely a cost benefit analysis that should be considered before doing a double reset of the krbtgt password as described below. You can certainly now detect a golden ticket if it is in use in your environment. However, as I’ve noted, you’ve already been compromised and have likely lost quite a bit. The golden ticket is primarily used to re-enter an environment without needing to steal credentials all over again.

This brings us to the detection stage.  The easiest way to detect if one is in your environment is if it is used after you’ve done the double reset of the Kerberos password. This will generate an event ID of 4769 on a domain controller with a failure code of 0x1F.  Creating a rule in SCOM for this event is very easy to do (see screenshot below), but keeping it from generating noise is the part that needs to be planned out.

The reason for this is that plenty of legitimate tickets can generate 0x1F failures if the double reset happens too quickly, and our goal is not to be concerned about the legitimate tickets, but to respond when we find a ticket that is a golden ticket. As such, you need to do a Kerberos reset very carefully if you want to detect the bad guy.  By default, our 4769 rule is not going to generate any failures in a normal operating environment.  Once we start with the password resets though, this can get very noisy if not done carefully as it will break all existing tickets until new ones are obtained.

• Step 1:  Obtain the default expiration timeline.  This same policy will give you the clock skew information (more on this in a bit). Again, by default this is 10 hours, but you may have a GPO that sets it to a different time frame, and you NEED to know the timeframe. 
• Step 2:  Force all passwords to be changed and do the first krbtgt password reset. DONT DO THIS UNTIL YOU UNDERSTAND THE EFFECT ON YOUR ENVIRONMENT.  Remember, that if an attacker has compromised accounts, they don’t need a golden ticket. This just lets them back in at some point after the initial compromise. If you’re dealing with an active recovery, you’re going to want to also remove internet access along with any other means that the attacker may have to enter the environment.  Otherwise, they would be able to potentially generate a new key during the waiting period. In this scenario, you may want to simply do a quick double reset, but do understand that every legitimate ticket will now fail and generate a 4769 with a 0x1F failure.  That will generate a lot of noise. The purpose of golden ticket monitoring, however, is to ensure that the bad guy doesn’t re-enter at some point down the road.  If you do a rapid reset, I would not have this rule turned on.  You can turn the rule on after this has happened and you’ve waited enough time to allow for existing tickets to regenerate.
• Step 3:  Wait, but not long.  A couple things need to happen before this rule will work properly and not generate any noise. 
  • Active Directory needs to replicate across all Domain Controllers.
  • Once replication is complete, you need to wait for tickets signed with the old key to get new tickets issued with the new key.  You obtained that value in step 1.
  • You need to also account for time skew. This is a smaller waiting period, maybe 15 minutes or so, but this can come into play.  The default clock skew is 5 minutes, so waiting twice that will account for potential clock skew issues.
  • Please note that at this step, you are still vulnerable to a golden ticket attack.
• Step 4:  Reset the krbtgt password again.  Remember that an attacker created a ticket that may not expire for years. They are hopefully not active in your environment when you perform this process.  Also, you need to remember that if they still have your existing credentials, they don’t need a golden ticket.  If you are not careful in following the steps, the only thing accomplished is giving yourself a false sense of security.   After this is done properly, when they attempt to use the golden ticket, the rule you created will generate an alert when a forged ticket is used.

Unfortunately, what this is telling us is that your environment was compromised at some point. If you’re doing this because of an active recovery, in that sense you already know it. However, if you’ve done this as a preemptive means of removing an attacker, you now have further investigation to do to see what was stolen previously.

Disclaimer: I am not an active directory expert, and while I’ve done my fair share of AD administration, this is not something I’ve personally done, and I would not recommend taking it lightly. I strongly recommend something like that be done under the watchful eye of someone who knows AD at a far better level than a standard administrator. You will want to use a trusted advisor. Microsoft does provide instructions on how to do this. Doing this properly will mitigate against most of the risks associated with it, and you will of course want to have validated that your backups work prior to doing this. Following the procedure carefully should mitigate against most of the risks associated with it, but I do recommend you make it a point of reading up on all of this and making sure you have thoroughly discussed this with Microsoft support or someone who has done this procedure before.

• PowerShell script to reset krbtgt password.
• Blog article on subject.

As you can see, this will help you detect golden tickets, but it is not exactly a non-invasive solution. Solutions such as Advanced Threat Analytics can actively detect Golden Tickets without the need to reset the Kerberos password; however, that does not mean that it would be wise to ignore this as a component to your security practice.  My next step will be to see if I can figure out how to use SCOM to detect this in a non-invasive manner.  I’m not sure if that is possible, but for now, this will work.

I’m working on a management pack that contains these monitors along with additional security rules along with other security related items that SCOM can provide. My only test environment is my lab, which is hardly a production grade environment. I do welcome feedback. While I can only support this to the best of my ability, I can provide it if this is something you are interested in trying it out. My main goal is to keep the noise down to a minimum so that each alert is actionable. While that is not always easy to do, trying this out in other environments will go along way to getting it to that point.  If this is something you are interested in testing, please hit me up on linked in or have your TAM reach out to me internally.