Freigeben über

Troubleshooting account lockout the PSS way

  • Artikel

I‘ve been thinking for some time about pulling together the typical approaches we use when troubleshooting account lockout issues.

So... here is the CSS/PSS approach to troubleshooting Account Lockouts.

#1 - Look at the Account Lockout Threshold policy that is defined for the Domain.
Applications commonly do several retries of logons if the first logon fails (in case the first or second failures were because of transient network conditions - which is not unreasonable behavior). This means that a user that enters an incorrect password ONCE can be locked out if the lockout threshold is low – depending on how the application behaves (how many retries it attempts).
Anything below 10 is just asking for helpdesk calls.

I.e. is this simply a result of a too narrow lockout threshold policy or is this a real issue?

#2 Determine how many users are being affected
The easiest way to determine the extent of the problem is to enable Netlogon debug logging on the PDC emulator DC (All other DC‘s will contact it for confirmation in case of an incorrect password).
Once it is enabled, allow it enough time (30-60 minutes or longer) to gather information from users attempting to log on and then copy the netlogon.log and netlogon.bak files to a location where you can examine them closer.

I.e. is this an outbreak of account lockouts that affects all users or is it just affecting a handful of specific users or a single user?
What is the common factor?

I usually run the following against the Netlogon file to filter out the lockouts:
type netlogon.log |find /i "0xC000006A“ >badpassword.txt
type netlogon.log |find /i "0xC0000234“ >lockedout.txt

This should give you a list of users which are supplying bad passwords and users who have been locked out.

Note: you can't do this against a live Netlogon.log file because of the running Netlogon service - simplest solution is to copy it elsewhere and run the commands against the copy.

#3 Look at which machines the bad password attempts are coming from If you see logon attempts for different users coming from the same machine it can mean that it‘s a multi-user machine (like a Terminal Server). It can however also mean that it has some suspect application running on it – in either case you should investigate closer.

#4 Drill down to which applications are sending the bad passwords

A network trace from the client or just examining which applications and service are running on it and stopping each in turn to isolate the issue will usually be enough.

TCPView from Sysinternals or Netstat are also good for this kind of investigation, matching the process ID of a service or application that creates a socket connection with a bad password attempt in the Netlogon log of a DC.

[Footnote: Windows 8 and Windows Server 2012 log the process ID of the application logging the lockout in the Netlogon log - which makes it much easier to troubleshoot this type of problme on W8/W2k12]

Common contributors can be OS components like Credman with stale passwords, services running under a specific domain account, dumb applications with insufficient retry logic, etc.

The Conficker virus was also notorious for attempting brute force password attacks against members of the built-in Administrators group in the Domain.

 

Note also that if you have a mixed environment you may get Account Lockout issues when you change passwords on one OS (client-side or DC-side) and then move to another legacy client that doesn't understand the protocol or algorithm used.

  

Summary:
- If you enable a lockout policy on the domain – you will get accounts locked out at some point.

- The lower the lockout attempt threshold – the more frequent the lockouts obviously.

- Using an account lockout policy is exposing yourself to a potential Denial of Service attack – there‘s nothing that prevents one spiteful user from attempting to log on as another user and entering an incorrect password until they‘ve locked out the account.

The potential value of using a lockout policy in your domain is to prevent brute-force password attacks. At the same time it exposes you to Denial-of-Service attacks if you enable it.

 When I started at Microsoft some years ago I came from a mixed support environment, during that time I had learned to love two extremly powerful string parsing utilities; GAWK and SED.  

I wrote a simple batch file that uses those two commands to parse the Netlogon file into a more visually pleasing format, here is a sample output from it:

 

 Summary of Netlogon log file Netlogon.bak
Netlogon log file starts at 07/14 17:07:40
....................ends at 07/15 09:09:19
-----------------------------------------------
Total successful logons of users: 30271
Total locked out logons of users: 475
Total logons of users with bad password: 16596
Total logons of users with unknown username: 2
Total logons from undefined subnets: 2
Total Access Denied errors: 0
Total trust password errors: 0
Total KDC PAC verifications : 1137

 ......

 #of attempted logons with bad passwords from each machine
----------------------------------------------------------------------------------
11430 DOMAIN\USER9424 from WORKSTATION-43928 (via EXCHANGE09)
2647 DOMAIN\USER9323 from WORKSTATION-44367 (via EXCHANGE14)
756 DOMAIN\USER5254 from WORKSTATION-44791 (via EXCHANGE09)
380 DOMAIN\SVCACCT01 from EXCHANGE09 (via EXCHANGE09)
380 DOMAIN\SVCACCT02 from EXCHANGE01 (via EXCHANGE01) 
  

In this log:

 -the top 3 users had a bad application running on their workstation which attempted to log on to their Exchange mailbox every 5 seconds if an incorrect password had been entered.  
- the passwords had been changed twice for service accounts running for an antivirus agent on EXCHANGE01 and EXCHANGE09, but the password hadn‘t been updated to reflect that on the servers.

 

Account Lockout and Management Tools
http://www.microsoft.com/downloads/details.aspx?familyid=7AF2E69C-91F3-4E63-8629-B999ADDE0B9E&displaylang=en  

Account Lockout Best Practices White Paper
http://www.microsoft.com/downloads/details.aspx?FamilyID=8C8E0D90-A13B-4977-A4FC-3E2B67E3748E&displaylang=en&displaylang=en

 

Troubleshooting Account Lockout
http://technet.microsoft.com/en-us/library/cc773155(WS.10).aspx

Account Lockout Tools
http://technet.microsoft.com/en-us/library/cc738772(WS.10).aspx

Comments

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    A brute password attempt involves millions of attempted logons - the difference between a lockout threshold of 5 and 50 is marginal in that context. I.e. you're not going to brute force a password with 45 extra attempts unless it's an unusually bad password in the first place.

  • Anonymous
    January 01, 2003
    You need to specify the name of the netlogon.log or netlogon.bak file that you're trying to analyze. From the error you're getting it looks like you're running the batch file while specifying the running netlogon.log file which is going to be locked. The batch file cannot be run on an active netlogon.log file, you'll need to copy the netlogon.log file and run the batch file against the copy. Note that the format of the Netlogon logs tends to change over time as the developers add or remove entries to the log....this means that some slight tweaking of the batch file may be necessary to match this to the machine you're analyzing netlogon logs from.

  • Anonymous
    January 01, 2003
    Try typing the commands in manually, it looks like you have copy/pasted the command from jwheelock's comment into a command prompt, this sometimes plays havoc with the quotation mark (") at the receiving end. I.e. if you look at "0xC000006A“ in the syntax you're using you'll see that the first quotation mark is different from the second - the first one is the correct format and the second one should be identical. ...and also make sure you copy the netlogon.log file to a different folder, you cannot run this against a live Netlogon.log file.

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    October 06, 2009
    The comment has been removed

  • Anonymous
    March 26, 2010
    The comment has been removed

  • Anonymous
    April 21, 2010
    2008 Domain Controller (64-bit) SP2:

  1. Enabled netlogon debugging with "nltest /dbflag:0x2080ffff"
  2. Manually search the netlogon.log for "0xC0000234" and I get hits, but when I try to run "netlogon.log |find /i  "0xC0000234“ >lockedout.txt" I get the following error: "the process cannot access the file because it is in use by another process"
  3. Tried disabling symantec 11.5 AV but no change. Ideas to get past the error? thanks,  jw

  • Anonymous
    July 30, 2010
    Hi, Excellent BLOG! Can you explain what you mean with marginal and why in: "The difference in the protection against brute force attacks that a lockout policy of 50 offers when compared with a lockout policy of 5 is marginal. "?

  • Anonymous
    August 08, 2011
    The comment has been removed

  • Anonymous
    October 04, 2012
    Ingolfur, re "you're not going to brute force a password with 45 extra attempts". How about a large AD deployment with hundreds of domain controllers and a more sophisticated attack that tries to hit each domain controller (say with a targeted LDAP simple bind attempt) with 49 password attempts within the account lockout observation period. With 500 DCs could you approach 24,500 logon attempts then without triggering an account lockout condition on any one of those DCs? I've not found a documented provision in native AD (or ADDS as it's now called) to pool logon failures across DCs for calculating the threshold.

  • Anonymous
    December 05, 2012
    Hi Ingolfur Arnar Stangeland                                              Excelent Script thanks a lot...

  • Anonymous
    December 31, 2013
    Pingback from A change to the fields in the Netlogon.log file from Windows 2012 and above

  • Anonymous
    December 31, 2013
    Pingback from A change to the fields in the Netlogon.log file from Windows 2012 and above