What happens when a group is deleted
A Critsit from a large enterprise customer came in the other day, problem description was as follows:
We’ve deleted a test group that contained 25000 users, now our 3rd party login script which looks at group membership for users and performs action based on which groups they are in is failing when it encounters the group in the Deleted Objects folder. This looks different per user and per DC, i.e. some users on some DC’s are not members of the deleted group but other users are listed as being members of the group that now has the DEL+GUID RDN and is located in the Deleted Objects container.
First thing that happens when the group is deleted is that it is moved to the Deleted Objects container in AD, this is normally not visible even to Admins but can be viewed with LDP.
The isDeleted attribute of the object is then set to True on the deleted group and routines to strip most attributes from the group object are called on it. Only a few attributes are kept, like the Sid, GUID, SidHistory and ACL’s.
The most interesting of the attributes that are stripped away is the ‘Members’ attribute. The Link Cleaner part of the Garbage Collection routines goes through the list of forward links in the 'Members' attribute and removes them, this is called immediately on the object when it is deleted (as opposed to every 12 hours for normal Garbage Collection).
The initial call removes up to 1000 forward links (members) from the members attribute of the group, if there were more than 1000 members of the group the link cleaner routine then schedules the remainder of the links to be removed in 60 seconds.
The garbage collection routine in general is a low-priority thread running within LSASS and defers to other more critical real-time sensitive tasks that LSASS is doing like user authentication and such.
This means that if other more CPU-intensive threads are running within LSASS on the DC then garbage collection may be delayed until such a time that the DC is under less load.
Another item of note is that the garbage collection occurs per DC and is not replicated, you can therefore have different results between different DC's if one of them happens to be heavily loaded to the point where LSASS is considered to be 'busy'. I.e. one DC will remove all the members but the 'busy' DC will only remove the first 1000 and delay the removal of the rest until it becomes less busy.
Monitoring this process is possible by setting HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesNTDSDiagnostics6 Garbage Collection to 4.
At this severity, you will see NTDS Database 1697 event with category Garbage Collection being logged into the Directory Service event log for each item the link cleaner removes a value from an object being cleaned. During a bulk deletion this will of course cause massive amount of data being logged which will quickly fill up the DS event log.
In the customer’s case what had happened was that some process on a client (or clients) was bombarding the PDC emulator DC with incorrect or expired password attempts (and probably at least one other DC) which caused the link removal of the remaining members of the deleted group to be continuously deferred for 60 seconds on those DC’s as LSASS was busy with handling the authentication attempts. The 3rd party login script was unable to handle the DEL:+GUID name of the deleted group which caused it to fail when the group name was encountered.
Once the client(s) were eliminated from the picture LSASS proceeded with removing the rest of the links 60 seconds later.
On a side note; an interesting trick you can use to quickly recover group membership is to use the AD Database Mounting tool in Windows Server 2008.
Using this trick you must first reanimate the deleted group in your target AD and then run an LDIFDE dump against the mounted AD database snapshot to recover the group membership.
This won't help in retrospect unfortunately, you'll need to have taken a snaphot earlier to use this or have scheduled snapshots to occur.
References:
How the Data Store works: http://technet.microsoft.com/en-us/library/cc772829.aspx
AD DS: Database Mounting Toolhttp://technet.microsoft.com/en-us/library/cc753246.aspx
How to restore deleted user accounts and their group memberships in Active Directoryhttp://support.microsoft.com/?id=840001
Viewing deleted objects in Acrive Directory: http://support.microsoft.com/kb/258310
Reanimating Active Directory Tombstone objects: http://technet.microsoft.com/en-us/magazine/cc137800.aspx
Searching for Deleted Objectshttp://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/distrib/dsbc_nar_cxyn.mspx?mfr=true
isDeleted attribute
http://msdn.microsoft.com/en-us/library/cc220036.aspx
A short explanation on the Garbage Collection Process of Active Directory
http://support.microsoft.com/kb/555994