NTDS Replication error 2095 and Virtual Server (long post)
Well, I broke down and bought a new workstation last weekend. It is a dual core AMD64 with 2 gigs of RAM and a 300 GB SATA drive and a 19 inch flat panel. Nice box! I lived with a Dell Precision 610 for about 4 years so it was definitely time for an upgrade. I started transferring files from the old workstation to the new one. I decided to migrate my Win2k3 Replica DC running in Virtual Server from the old workstation to the new as well. Since I was actually logging onto the local "old workstation", I shut down the "virtual" domain controller and stopped the Virtual Server service. I then made a remote desktop connection to the old from the new and started copying files... I had shares open from the old to the new and from the new to the old...copying about 10GB of pictures and about 7 GB of songs...life is good. Well, I had some serious disk space problems on the old box and I decided to delete the "offline backups" of my Virtual Server (VHD/VMC files). Life is still good. After all the files were copied off the old box, I decided to fire up my replica DC in Virtual Server...but it did not start. I checked the Virtual Server management interface and there was an error "file not found". Uh oh...I deleted the actual working files! Well, a quick search for *.vhd on that box produced nothing. I then looked on my SBS box and found a pair of VHD/VMC files dated 12-23-2005. I made that backup when blogging my Migration Scenario. Good, well within the tombstone period. I was lucky! I then install Virtual Server 2005 on my new box, copy the files over and boot her up. She boots up fine, life is good again. Whew... Then just to be sure, I go check event viewer on my recently restored "virtual" domain controller and am presented with the lovely set of event below:
Event Type: Error
Event Source: NTDS Replication
Event Category: (5)
Event ID: 2095
Date: 2/3/2006
Time: 6:50:44 PM i
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: WIN2KDC
Description:
During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers.
Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC.
If not resolved immediately, this scenario will result in inconsistencies in the Active Directory databases of this source DC and one or more direct and transitive replication partners. Specifically the consistency of users, computers and trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data may vary, affecting the ability to log on, find objects of interest and perform other critical operations.
To determine if this misconfiguration exists, query this event ID using https://support.microsoft.com or contact your Microsoft product support.
The most probable cause of this situation is themproper restore of Active Directory on the local domain controller.
User Actions:
If this situation occurred because of an improper or unintended restore, forcibly demote the DC.
Remote DC:
7e3836bf-dbd3-4c43-80d7-679ec27932c8
Partition:
DC=company,DC=local
USN reported by Remote DC:
472083
USN reported by Local DC:
369571
For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.
Event Type: Error
Event Source: NTDS General
Event Category: (12)
Event ID: 2103
Date: 2/3/2006
Time: 6:50:44 PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: WIN2KDC
Description:
The Active Directory database has been restored using an unsupported restoration procedure.
Active Directory will be unable to log on users while this condition persists. As a result, the Net Logon service has paused.
User Action
See previous event logs for details.
For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.
Event Type: Warning
Event Source: NTDS General
Event Category: (5)
Event ID: 1113
Date: 2/3/2006
Time: 6:50:44 PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: WIN2KDC
Description:
Inbound replication has been disabled by the user.
For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.
Event Type: Warning
Event Source: NTDS General
Event Category: (5)
Event ID: 1115
Date: 2/3/2006
Time: 6:50:44 PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: WIN2KDC
Description:
Outbound replication has been disabled by the user.
For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.
Event Type: Warning
Event Source: NTDS General
Event Category: (9)
Event ID: 1173
Date: 2/3/2006
Time: 6:50:44 PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: WIN2KDC
Description:
Internal event: Active Directory has encountered the following exception and associated parameters.
Exception:
e0010002
Parameter:
0
Additional Data
Error value:
8451
Internal ID:
108132e
For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.
So now what?? Clicking on the "links" in the events took me to these articles: https://support.microsoft.com/kb/875495/en-us and https://support.microsoft.com/kb/885875/en-us. Anytime you seen an article with "USN Rollback" in the title, smoke 'em if you got 'em. I alway have to go consult with Mark Stanfill (from Inside SBS fame, https://blogs.technet.com/sbs) when someone even mentions USN Rollback. Ok, those are scary events. I immediately shut down the "virtual" replica domain controller and boot up into Directory Services Restore Mode and go check my backups. YES, I even do backups of my "virtual" domain controller. I have one from 1-30-2006. I kick off NTBACKUP, browse over to my old workstation (where all my backups go), load up the BKF file and do a system state restore. 15 minutes later, I reboot and am presented with this event in the Directory Service event log:
Event Type: Information
Event Source: NTDS Replication
Event Category: (5)
Event ID: 1109
Date: 2/3/2006
Time: 7:16:30 PM
User: N/A
Computer: WIN2KDC
Description:
Active Directory has been restored from backup media, or has been configured to host an application partition. The invocationID attribute for this domain controller has been changed. The highest update sequence number at the time the backup was created is as follows.
InvocationID attribute (old value):
367df4eb-81d5-4903-b533-6cdf4510919f
InvocationID attribute (new value):
61ebb45e-641d-4872-907c-caf57b286c5b
Update sequence number:
462230
The invocationID is changed when a domain controller is restored from backup media or is configured to host a writeable application directory partition.
For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.
Life is good again! This was NOT luck, I had planned for this. If you are using Virtual Server in a production environment, please...please...please...please read this article and this article AND start doing system state backups on your virtual servers as well. It is NOT good enough to simply do an offline backup of the VHD/VMC files (meaning shut down the virtual server and copy the files). My practice is to do a system state twice a week and an offline backup once per month. Of course on my SBS box, backups are daily. With those three, I sleep well at night knowing my 2 user environment is safely backed up and restorable. Also, the above errors could also happen with "other" types of backups (insert GHOST). I love creative backup solutions! Make sure they work! Test your restore procedure. Test it again! In summary, I kept my replica intact with the built-in software (NTBackup) and a 470 MB file on an IDE drive. Anyone checked the prices of drives lately?
Life is good again...now I can go watch all those movies I recorded with my new Media Center PC <while I was restoring a domain controller that was running on the same box in Virtual Server>!
Petergal