Bulk Password Resets

When a user resets their password, what happens? What about if ALL your users reset their password?  Can your infrastructure handle it?  Are there "special" changes that you'd want to make?  More importantly, this is probably such an edge case that it's not on your top 150 list of things to do?  Well, just for kicks, let's look at bulk password resets.

Consider the following scenario:

For whatever reason floated their boat that day, your security team wakes you up in the middle of the night with a mandate that all 20,000 users in your domain need to be forced to change their passwords when the log in the next morning.  Your first thought, is probably something about finding a script that can expire the passwords on all of these accounts, but at some point it's likely going to occur to you that the majority of these 20,000 users are all going to come walking into their offices around the same time (8am, 9am, whenever...) and try to log in.  Can your DC's handle it?

It turns out that this was a scenario that we wanted to test (without the security boat floating) in January 2003 while we were dogfooding Windows Server 2003.  Recently the topic came up again in a discussion, so I dusted out the original report that was written to refresh my memory of what we saw and what we'll do if we ever need to do a mass password reset.

To start with, first you have to assess the true impact.  If you're users are spread out across many time zones, then "next morning" is really a relative thing and you should start to consider if any of users are really in the middle of their day.  Those are the folks that are going to call help desk on you.  In our case, we intentionally reset the passwords of our users here in Redmond, this ensured that nearly everyone was going to come in around the same time (10am'ish) and reset. 

Before we get to far into this though, let's look at password reset behavior.  When a user logs on and changes their password, there is an extra step that occurs.  The NetLogon service of the DC making the password change forwards the new password to the PDC for the domain.  Generally this is a good thing, because if the user attempts to authenticate using their new password against a DC that hasn't replicated it in yet, the "failed" password is checked against the PDC for the domain and the user will succeed.  Both of these behaviors, the password update and the additional checks, can present a problem during a bulk reset because the volume of changes being pushed from many DC's to one PDC can melt your PDC into slag on the data center floor...and that's always messy to clean up.  So the solution is to hide your PDC from the sudden flood of traffic that's going to come from the DC's.  

To solve this problem, there is a registry key called AvoidPdcOnWan that needs to be set on every DC (FOR loop with "reg add" is your friend here).  Setting this key tells every DC to let normal AD replication take care of getting the password to the PDC.  Also, if presented with a bad password, the DC won't check with the PDC but instead will just fail the authentication.  The second step is to move the PDC into it's own AD site linked with a site link.  In our case, all of the DC's for these 20K users were in a single AD site, which included the PDC.  So we created a new site called PDC and linked it to our main site with the minimum replication interval.  This effectively put the PDC "on the WAN" to all other DC's.

The net result of these changes is that the bulk of your DC's will do the work AND they'll play nice with the PDC for the domain (always a good thing).  Unfortunately there is the behavior change that the DC's won't check a "bad" password with the PDC, so you wouldn't want to leave this config in place all the time as that's a "bad user experience" which normally translates into help desk calls. 

Hopefully you never have to reset the passwords for a large bulk of users, but if you do, then setting this reg key, creating a new site, and moving the PDC into it will at least save you from the secondary thrash of watching your PDC slowly fall onto it's side and beg for mercy.