Windows machine account passwords and VM snapshots
Windows machine account passwords and VM snapshots
The problem
Almost every virtualization platform like Hyper-V provides ways to capture the complete state of a VM as a snapshot. You can later restore back to this snapshot quickly to get back to what you are doing earlier. This provides a powerful technique with many uses: changing and testing applications, prepare demo setups, create quick backups, to name a few.
However, if the VM happens to be joined to an active directory domain, after restoring a snapshot, you occasionally find that all authentication involving the VM seem to fail. From this VM, you cannot access any network shares outside. From your other machines, you cannot access any files shared in this VM, etc. You cannot even login as yourself, a domain user, into the VM: you get an error like “Windows cannot connect to the domain, either because the domain controller is down or otherwise unavailable, or because your computer account was not found. Please try again later.”
What is happening here?
The answer is machine account password mismatch: the VM thinks its machine account password to be something, while the domain controller believes it to be something else. The VM cannot authenticate itself to the domain controller.
In this blog, I will describe some simple experiments to understand problem better and suggest few strategies for dealing with it.
Machine account passwords
Just like user account passwords, machine account password is a secret using which a Windows domain member authenticates itself to the domain controller and establishes a secure channel.
When the computer is started, a service called NetLogon uses the machine account password and tries to establish a secure session with the domain controller. Other services running on this computer with LocalSystem or NetworkService credentials require this authenticated secure channel to get access to domain resources. The usual CTRL+ALT+DEL Winlogon process also relies on this authenticated secure channel to send user credentials to the domain controller for verification and log them into the computer.
The password is first created when the computer is joined to a domain. It is shared by domain controller and the computer. After that, for security reasons, the computer at periodic intervals (usually 30 days) negotiates with the domain controller and changes its machine account password. After this change, both the domain controller and the computer use the new password for authentication.
While using snapshots, when the domain member is restored to an older snapshot, it loses track of any password change changes done later and tries to use an older password. Hence it fails to authenticate itself. Various things fail as a consequence.
Experimenting with machine account passwords and VM snapshots
In this blog, I will describe a simple experiment to force machine password problems and analyze it. For this purpose, we will need a Windows VM that is a member of some domain, and hosted in Hyper-V.
In order to study and manipulate machine passwords, we will use a tool called nltest.exe. This is available as part of the Windows Support Tools. This needs to be downloaded and installed. Nltest.exe uses Windows API like I_NetLogonControl2 underneath and interacts with Netlogon service.
We will login as administrator, launch a command prompt, and then run the following commands:
(1) Find out the domain the computer is part of.
>> nltest.exe /parentdomain
mydomain.corp.mycompany.com. (1)
The command completed successfully
(2) Since we have already logged in, this computer would have authenticated with its domain controller and established a secure channel. Verify the health of this channel.
>> nltest.exe /sc_query:mydomain.corp.mycompany.com
Flags: 30 HAS_IP HAS_TIMESERV
Trusted DC Name \\HYD-FE-DC-02.mydomain.corp.mycompany.com
Trusted DC Connection Status Status = 0 0x0 NERR_Success
The command completed successfully
(3) At times, it may be possible that computer is using a cached session channel which may not be valid anymore. Just to be sure, force the computer to re-authenticate with its domain controller and re-establish a new channel.
>> nltest.exe /sc_reset:mydomain.corp.mycompany.com
Flags: 30 HAS_IP HAS_TIMESERV
Trusted DC Name \\hyd-fe-dc-01.mydomain.corp.mycompany.com
Trusted DC Connection Status Status = 0 0x0 NERR_Success
The command completed successfully
The new secure channel has gotten established correctly as well.
(4) Now look at the machine passwords the machine is actually using. For security reasons, the actual password is never shown. We can supply a message that will be hashed with the password and the digest will be displayed.
>> nltest.exe /cdigest:MESSAGETOHASH /domain:mydomain.corp.mycompany.com
Account RID: 0xce947
New digest: 50 40 f1 b1 9d 2d 0f 80 dc 46 e4 78 a7 ee 43 e9 P@±¦¥-.Ç_FSxºeCT
Old digest: 50 40 f1 b1 9d 2d 0f 80 dc 46 e4 78 a7 ee 43 e9 P@±¦¥-.Ç_FSxºeCT
The command completed successfully
There is one interesting thing to note here: There are two digests shown corresponding to two different passwords. For sake of reliability, the computer always remembers its last two machine passwords: the current one and the preceding one. This is necessary because a password change may take time to propagate to all domain controllers in forest. So it is necessary at times to use the old password as well.
(5) At this point, take a snapshot using Hyper-V. Let’s call it Time1. We will use this snapshot later to reproduce password mismatch problems.
(6) Now, for the sake of the experiment, force a machine password change. Machine password changes are always initiated by the domain member (and not by the domain controller). In general, machine password changes are done in once in 30 days. But we don’t want to wait that long here and so take matters into our hands.
>> nltest.exe /sc_change_pwd:mydomain.corp.mycompany.com
Flags: 0
Connection Status = 0 0x0 NERR_Success
The command completed successfully
(7) Look at the new passwords used.
>> nltest.exe /cdigest:MESSAGETOHASH /domain:mydomain.corp.mycompany.com
Account RID: 0xce947
New digest: 27 ef e6 b9 f3 24 e9 8a 17 a0 c2 f3 12 28 5e ca 'nµ¦=$Tè.á-=.(^-
Old digest: 50 40 f1 b1 9d 2d 0f 80 dc 46 e4 78 a7 ee 43 e9 P@±¦¥-.Ç_FSxºeCT
The command completed successfully
As you can see, the previous digest 50* has become the old one, and a new digest 27* is the current active password.
(8) Since last two passwords are tracked, to force a problem we will need to change the password once more.
>> nltest.exe /sc_change_pwd:mydomain.corp.mycompany.com
Flags: 0
Connection Status = 0 0x0 NERR_Success
The command completed successfully
(9) Look at the new passwords again
>> nltest.exe /cdigest:MESSAGETOHASH /domain:mydomain.corp.mycompany.com
Account RID: 0xce947
New digest: 13 b1 5f 2b ac b0 35 43 cc 00 81 d8 bf 68 5e 76 .¦_+¼¦5C¦.ü++h^v
Old digest: 27 ef e6 b9 f3 24 e9 8a 17 a0 c2 f3 12 28 5e ca 'nµ¦=$Tè.á-=.(^-
The command completed successfully
Now, the original digest 50* that was used in our snapshot Time1 is gone.
(10) Now go to Hyper-V and restore to snapshot Time1.
(11) Now, check out the machine account password used here.
>> nltest.exe /cdigest:MESSAGETOHASH /domain:mydomain.corp.mycompany.com
Account RID: 0xce947
New digest: 50 40 f1 b1 9d 2d 0f 80 dc 46 e4 78 a7 ee 43 e9 P@±¦¥-.Ç_FSxºeCT
Old digest: 50 40 f1 b1 9d 2d 0f 80 dc 46 e4 78 a7 ee 43 e9 P@±¦¥-.Ç_FSxºeCT
The command completed successfully
Since Hyper-V has restored the snapshot, the computer believes its machine passwords to 50*. Obviously this does not match what the domain controller thinks, since the password was changed a little while back.
(12) Now get the computer to re-authenticate with the domain controller
>> nltest.exe /sc_reset:mydomain.corp.mycompany.com
I_NetLogonControl failed: Status = 5 0x5 ERROR_ACCESS_DENIEDS
Voila!
As expected, Domain controller has refused to establish a secure channel. Try to access some network shares from here now. Or logoff and try to login again as a domain user. All fail. This VM is now busted. It cannot authenticate itself to the domain controller.
Strategies for dealing with machine account password problem
There are few broad strategies to deal with machine account password problems that I am aware of:
(1) Increase machine account password age or disable password changes altogether: Both these can reduce likelihood of the problem. These settings are available on the domain member (and not in the domain controller). As such, you can change them on your computer. However, some domain administrators frequently use machine account password age to run scavenging scripts on the domain controller. When this happens, your machine can get knocked off the domain and you are in trouble.
(2) When you restore a snapshot, detect if password has expired. If yes, then remove the machine from the domain and join it back. You will need a privileged domain account to do this. Then you can start using the machine.
Detect the problem
>> nltest.exe /sc_reset:mydomain.corp.mycompany.com
I_NetLogonControl failed: Status = 5 0x5 ERROR_ACCESS_DENIEDS
Correct it by removing and joining it back to the domain. You can use the netdom.exe tool from support tools, or you can do this from windows explorer.
>> "netdom.exe" remove frodo50 /Domain:mydomain /userd:mydomain\sudhakar /passwordd:***************
The command completed successfully.
>> "netdom.exe" join frodo50 /Domain:mydomain /userd:mydomain\sudhakar /passwordd:***************
The command completed successfully.
Reboot the VM. Now it will work well.
(3) The final strategy is a bit sophisticated. Create your own domain controller VM and host it alongside the domain member VM you are using. Snapshot and restore both of them together so that there is never any mismatch. As a bonus, since you have your own domain controller, there are a lot of other powerful things you can do. The product I work on, Visual Studio Lab Management 2010, provides a feature called Network Isolation to make this process easier.
N. Sudhakar
--------------------------
Views and opinions expressed in this blog are my own and do not reflect that of my employer.
Comments
Anonymous
November 01, 2010
Thank you for the excellent blog. It explained exactly to me why I was seeing that error. I am in a test environment so I have a large tree of VM's that are now broken because of a foolish step I took. I don't want to remove and join them everytime I restore them and I don't want to rebuild the tree of VMs. Is there a way to get the machine account password from my old VMs and send that password to the domain controller as my desired password?Anonymous
March 28, 2011
I want to say thank you for giving me a better picture of machine account passwords. We are on the way to deploying deep freeze and this was one of the stumbling blocks that we needed to get around.Anonymous
May 24, 2011
JoeCode, Did you find out a way to set the password on the DC?Anonymous
February 19, 2012
I'm dealing with this problem right now, so this was some useful information for me. Thank you!Anonymous
October 03, 2012
If you still have access to the registry before you restore your snapshop, you can grab the machine account password from it and then import them into the registry of the restored snapshot. I have done this successfully several times. If involved getting access to the keys via the system account. You can also automate the backup of these keys to another system, which I have also done so that I always have them ready.Anonymous
January 13, 2013
There's a mistake in this. You don't need to change the password twice as the DC only has knowledge of the current password (after replication has completed) reverting to an earlier snapshot will break the VM after a single password change.Anonymous
December 04, 2013
The comment has been removedAnonymous
March 12, 2014
Just wanted to comment that your suggestion of snapshotting and restoring a domain controller is a bad one. If not done in a purely isolated environment it can cause restoration of deleted AD objects, corruption of existing objects and other very bad things. Look up USN rollback for a description of this mess. Best to keep your DCs current and fix the machines IMO.Anonymous
March 17, 2014
NOTE: this is the fastest way I know to resync a VM after reverting; NETDOM RESETPWD /server:<Domain_Controller> /UserD * /PasswordD * /SecurePasswordPrompt It will prompt for your account and the resync (it then takes some seconds for it to sync to your logon server depending what server you used in above command) No reboot required.Anonymous
April 17, 2014
Hey Menvert I've been looking for a way to do this for a while now. I'm not working on resyncing a VM to a domain controller but I have been dealing with trust relationship error issues on Win7 thin clients. I've been trying to find a way to use a combination of local machine credentials and domain credentials to synchronize the machine local password and the domain password for AD computer accounts. The NETDOM command you provided did the trick, thanks!Anonymous
August 27, 2014Method 1
- Log in locally
- Run POWERSHELL as Administrator, then
- Test-ComputerSecureChannel –Credential DomainUser –Repair
- Done
- Method 2
- Log in locally
- Run POWERSHELL as Administrator, then
- Reset-ComputerMachinePassword [-Credential <PSCredential>]
- Reboot
- Anonymous
December 17, 2014
Hi , This third solution is too critical for domain environment because one of the domain controller snapshot will revert back and replicated entire configured domain. (3) The final strategy is a bit sophisticated. Create your own domain controller VM and host it alongside the domain member VM you are using. Snapshot and restore both of them together so that there is never any mismatch. As a bonus, since you have your own domain controller, there are a lot of other powerful things you can do. The product I work on, Visual Studio Lab Management 2010, provides a feature called Network Isolation to make this process easier.