Fighting Hyper-V Physical Host Time Drift – Utilizing NTP to Synchronize Clocks

My team at Microsoft has an engineering lab, one of which, I’ve spent a lot of time designing and building that provides our team some robustness.  We run our own “mini” lab for our engineering purposes that runs very much like a large number of enterprises.  By this, I mean we have our own self-hosted domain, DNS, DHCP, and application services such as ConfigMgr 2007, PKI, IIS, and SQL.  This lab started off running Windows Server 2008 Hyper-V physical hosts (Dell PE2950) that were then upgraded (in-place) to Windows Server 2008 Beta and RC1.  Recently, these physical hosts were cleaned and upgraded to a fresh Server 2008 R2 RTM and we used VMM to migrate our VMs off the hosts prior to doing so.  The upgrade went well though there were some snags that I thought I just might share…

SCENARIO

You have n number of Hyper-V hosts running Windows Server 2008 with various VMs living on them that are “production” like.  Because RTM doesn’t offer Live Migrations, you will fail to be able to migrate VMs off the hosts without some amount of downtime though SAN transfers are much quicker.  When I can’t purchase new hardware but I want to take advantage of R2 features, how might I do this?

ANSWER

Over the next few days, I will spend sometime talking about this very scenario and how I did it as a way to maybe share with folks some coolness…

Kerberos & Time Drift – Like mixing Oil & Water

Issue #1:  I’ve started the migration and I’m starting to get complaints that virtual machines are offline or unavailable.  However, when checking the VMs on the physical hosts they are up and running and seem alive.  They can ping, etc. but no clients can utilize services offered on these machines.

We ran into this issue and the root cause was due to the fact that the Hyper-V hosts time was drifting.  The physical hosts are a member of a domain that is comprised of virtual machine’s that are the DCs which isn’t reality in organizations but due to this they (the DCs) didn’t have good time.  This problem occurs due to the default settings for the integration services that have the virtual machines get their time from the physical hosts where they are running. 

Due to this, you can disable this if you like but not one that I would recommend.

W32TIME SERVICE & AUTHORITIVE TIME SERVER

As many should already know, Kerberos authentication’s documentation for Windows says that a difference of 5 minutes by any two machines who attempt to authenticate using    Kerberos will fail.  The hosts were drifting utilizing the default settings for the W32time service is to use NT5DS (Active Directory).  This causes issues in some cases, as we found out.

 

This issue can be fixed with these steps on your physical hosts:

1. Open Notepad on the server
2. Copy the following into Notepad -

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time]
"DisplayName"="@%SystemRoot%\\system32\\w32time.dll,-200"
"ImagePath"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\
74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,73,\
00,76,00,63,00,68,00,6f,00,73,00,74,00,2e,00,65,00,78,00,65,00,20,00,2d,00,\
6b,00,20,00,4c,00,6f,00,63,00,61,00,6c,00,53,00,65,00,72,00,76,00,69,00,63,\
00,65,00,00,00
"Description"="@%SystemRoot%\\system32\\w32time.dll,-201"
"ObjectName"="NT AUTHORITY\\LocalService"
"ErrorControl"=dword:00000001
"Start"=dword:00000003
"Type"=dword:00000020
"ServiceSidType"=dword:00000001
"RequiredPrivileges"=hex(7):53,00,65,00,41,00,75,00,64,00,69,00,74,00,50,00,72,\
00,69,00,76,00,69,00,6c,00,65,00,67,00,65,00,00,00,53,00,65,00,43,00,68,00,\
61,00,6e,00,67,00,65,00,4e,00,6f,00,74,00,69,00,66,00,79,00,50,00,72,00,69,\
00,76,00,69,00,6c,00,65,00,67,00,65,00,00,00,53,00,65,00,43,00,72,00,65,00,\
61,00,74,00,65,00,47,00,6c,00,6f,00,62,00,61,00,6c,00,50,00,72,00,69,00,76,\
00,69,00,6c,00,65,00,67,00,65,00,00,00,53,00,65,00,53,00,79,00,73,00,74,00,\
65,00,6d,00,54,00,69,00,6d,00,65,00,50,00,72,00,69,00,76,00,69,00,6c,00,65,\
00,67,00,65,00,00,00,00,00
"FailureActions"=hex:80,51,01,00,00,00,00,00,00,00,00,00,03,00,00,00,14,00,00,\
00,01,00,00,00,60,ea,00,00,01,00,00,00,c0,d4,01,00,00,00,00,00,00,00,00,00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Config]
"FrequencyCorrectRate"=dword:00000004
"PollAdjustFactor"=dword:00000005
"LargePhaseOffset"=dword:02faf080
"SpikeWatchPeriod"=dword:00000384
"LocalClockDispersion"=dword:0000000a
"HoldPeriod"=dword:00000005
"PhaseCorrectRate"=dword:00000001
"UpdateInterval"=dword:00007530
"EventLogFlags"=dword:00000002
"AnnounceFlags"=dword:0000000a
"TimeJumpAuditOffset"=dword:00007080
"MinPollInterval"=dword:0000000a
"MaxPollInterval"=dword:0000000f
"MaxNegPhaseCorrection"=dword:ffffffff
"MaxPosPhaseCorrection"=dword:ffffffff
"MaxAllowedPhaseOffset"=dword:0000012c

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Parameters]
"ServiceDll"=hex(2):25,00,73,00,79,00,73,00,74,00,65,00,6d,00,72,00,6f,00,6f,\
00,74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,\
77,00,33,00,32,00,74,00,69,00,6d,00,65,00,2e,00,64,00,6c,00,6c,00,00,00
"ServiceMain"="SvchostEntry_W32Time"
"ServiceDllUnloadOnStop"=dword:00000001
"Type"="NTP"
"NtpServer"="time.windows.com,0x9"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Security]
"Security"=hex:01,00,04,80,84,00,00,00,90,00,00,00,00,00,00,00,14,00,00,00,02,\
00,70,00,05,00,00,00,00,00,14,00,fd,01,02,00,01,01,00,00,00,00,00,05,12,00,\
00,00,00,00,18,00,ff,01,0f,00,01,02,00,00,00,00,00,05,20,00,00,00,20,02,00,\
00,00,00,14,00,8d,01,02,00,01,01,00,00,00,00,00,05,04,00,00,00,00,00,14,00,\
8d,01,02,00,01,01,00,00,00,00,00,05,06,00,00,00,00,00,14,00,9d,01,02,00,01,\
01,00,00,00,00,00,05,13,00,00,00,01,01,00,00,00,00,00,05,12,00,00,00,01,01,\
00,00,00,00,00,05,12,00,00,00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\NtpClient]
"DllName"=hex(2):25,00,73,00,79,00,73,00,74,00,65,00,6d,00,72,00,6f,00,6f,00,\
74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,77,\
00,33,00,32,00,74,00,69,00,6d,00,65,00,2e,00,64,00,6c,00,6c,00,00,00
"Enabled"=dword:00000001
"InputProvider"=dword:00000001
"AllowNonstandardModeCombinations"=dword:00000001
"CrossSiteSyncFlags"=dword:00000002
"ResolvePeerBackoffMinutes"=dword:0000000f
"ResolvePeerBackoffMaxTimes"=dword:00000007
"CompatibilityFlags"=dword:80000000
"EventLogFlags"=dword:00000001
"LargeSampleSkew"=dword:00000003
"SpecialPollInterval"=dword:00000900
"SpecialPollTimeRemaining"=hex(7):00,00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\NtpServer]
"DllName"=hex(2):25,00,73,00,79,00,73,00,74,00,65,00,6d,00,72,00,6f,00,6f,00,\
74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,77,\
00,33,00,32,00,74,00,69,00,6d,00,65,00,2e,00,64,00,6c,00,6c,00,00,00
"Enabled"=dword:00000001
"InputProvider"=dword:00000000
"AllowNonstandardModeCombinations"=dword:00000001
"EventLogFlags"=dword:00000000
"ChainEntryTimeout"=dword:00000010
"ChainMaxEntries"=dword:00000080
"ChainMaxHostEntries"=dword:00000004
"ChainDisable"=dword:00000000
"ChainLoggingRate"=dword:0000001e

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\VMICTimeProvider]
"Enabled"=dword:00000001
"InputProvider"=dword:00000001
"DllName"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\
74,00,25,00,5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,76,\
00,6d,00,69,00,63,00,74,00,69,00,6d,00,65,00,70,00,72,00,6f,00,76,00,69,00,\
64,00,65,00,72,00,2e,00,64,00,6c,00,6c,00,00,00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\VMICTimeProvider\Parameters]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TriggerInfo]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TriggerInfo\0]
"Type"=dword:00000003
"Action"=dword:00000001
"GUID"=hex:ba,0a,e2,1c,51,98,21,44,94,30,1d,de,b7,66,e8,09

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TriggerInfo\1]
"Type"=dword:00000003
"Action"=dword:00000002
"GUID"=hex:6e,51,af,dd,c2,58,66,48,95,74,c3,b6,15,d4,2e,a1

1. Save it as useNTPTime.reg
2. Double-click on the registry file to import
3. Open a CMD prompt
4. Type net stop w32time && net start W32time

NOTE: This step modifies the registry. Please assume all normal precautions such as backing up the registry or exporting the original values as a method to roll back if something drastically fails.

Woo.. Life is good again.

-Chris

Comments

• Anonymous
  January 01, 2003
  Hey ABS- That is a great question.  In hopes of being very transparent and honest, I wasn't aware of that option you provided though I'm certainly aware now.  I really appreciate it and I will check into that as it does offer a more elegent solution. Thanks, -Chris

• Anonymous
  January 01, 2003
  Chris, why wouldn't you just use the w32tm commands to make this change? i.e.  w32tm /config /syncfromflags:MANUAL

• Anonymous
  January 01, 2003
  Hey Paul, Great suggestion and I realize that I didn't do that.  I apologize! This registry file does the following:

1.  Changes the NTPServer enabled to DWORD 1 (from default of 0)
2.  Changes the Parameters type for W32Time from default (NT5DS) to NTP This enables the type from NT5DS to NTP thus enabling the ability to all point to the same share.

• Anonymous
  August 31, 2009
  Chris, You may want to document the changes in your reg file that makes this a solution for the time drift issue.