Logon Failures Involving Virtual Machines in Windows Server 2012
Welcome back to the CORE Team blog. The General Availability date for Windows 8 and Windows Server 2012 has come and gone, and we here on the CORE Team expect more of you will be diving in and taking part in all of the excitement around these new products. To make sure you have a great experience, we endeavor, whenever possible, to make you aware of situations that may temporarily 'inconvenience' you. That is the purpose of this blog.
We have recently encountered several instances where a specific group policy configuration can affect the proper functioning of a Windows Server 2012 Hyper-V virtualization solution. This is due primarily to changes made in Windows Server 2012 Hyper-V functionality that will also be explained here.
The two scenarios we have seen thus far are:
- Virtual machines failing to start
- Virtual machines failing to live migrate
In both of these scenarios, the problem is the result of a logon failure. Here is an example of a pop-up error message you may see when a virtual machine fails to start.
The critical piece of the error message is, "Logon failure: the user has not been granted the requested logon type at this computer (0x80070569.) "
In the second scenario where a virtual machine fails to live migrate, an Event ID 21502 error message is registered in the Hyper-V-High-Availability log. The critical piece of the error message is, "Failed to create Planned Virtual Machine at migration destination. Logon failure: the user has not been granted the requested logon type at this computer (0x80070569.) "
Investigation of these events revealed that a custom Group Policy was modifying the user accounts that are allowed to Logon on as a Service on each Hyper-V server.
In Windows Server 2012, a special security group, NT VIRTUAL MACHINE\Virtual Machines is created when the Hyper-V Role is installed. Members of this group require the right to Create Symbolic Links (SeCreateSymbolicLinkPrivilege) and to Log on as a Service (SeServiceLogonRight). The SID associated with the group is S-1-5-83-0. The security group is maintained by the Hyper-V Management Service (VMMS). To ensure members of the NT VIRTUAL MACHINE\Virtual Machines security group maintain the rights they need, VMMS registers with Group Policy in order to update the local security policy whenever Group Policy is refreshed.
The NT VIRTUAL MACHINE\Virtual Machines group did not exist in previous versions of Hyper-V. As each virtual machine is started on a Hyper-V server, its account (Virtual Machine ID (VM_ID)) is added to the NT VIRTUAL MACHINE\Virtual Machines group and VMMS creates a Virtual Machine Worker Process (vmwp.exe). Examples of these processes are visible in Task Manager:
The VM_ID is the virtual machine account that is used to gain access to its own resources and prevent other virtual machines from gaining access to those same resources. As an example, if I run the following PowerShell command, it is easy to see the rights given to the virtual machine account to one of its resources (a virtual hard disk in this case):
Get-Acl -Path E:\Virtual Machines\Contoso-FS1\Virtual Hard Disks\contoso-fs1.vhdx | FL AccessToString AccessToString : NT VIRTUAL MACHINE\E57917F3-31C3-456E-B1BA-5E45B4CC7E0C Allow Write, Read, Synchronize BUILTIN\Administrators Allow FullControl NT AUTHORITY\SYSTEM Allow FullControl NT AUTHORITY\Authenticated Users Allow Modify, Synchronize BUILTIN\Users Allow ReadAndExecute, Synchronize |
Since the VMWP is an extension of VMMS, VMMS performs a service logon to create an access token that is used to run the VMWP. In order for this to work, the NT VIRTUAL MACHINE\Virtual Machines security group must be granted the Log on as a Service right. In previous versions of Hyper-V, the VMWP ran in the context of a different account, NETWORK SERVICE, which is an account defined by SYSTEM.
Windows Server 2008 R2 SP1 Hyper-V Server
To find out more information about the NETWORK SERVICE account, review this MSDN resource (https://msdn.microsoft.com/en-us/library/windows/desktop/ms684272(v=vs.85).aspx).
The error message, previously mentioned, refers to a 'user' not being granted a 'logon type'. That user, again as seen in Task Manager, is the Virtual Machine ID (VM_ID), and the logon type is 'Log on as a Service.'
Now that we understand the new changes, what needs to be done? A detailed Knowledge Base (KB) article was written in cooperation with the Directory Services team that provides additional details.
KB2779204
Starting or Live Migrating Hyper-V virtual machines may fail with error 0x80070569 on Windows Server 2012-based computers
https://support.microsoft.com/kb/2779204
Briefly, one of two things must happen:
- Hyper-V Administrators need to get with their Domain Administrators to review Group Policies to see if any involve specific user accounts being granted the Log on as a Service right, and, if so, have the policy modified appropriately
- Create an OU in Active Directory and place all hyper-V servers in that OU and block policy inheritance
Note: Option (2) is recommended by the Hyper-V Product Team
Tip: Administrators can temporarily, but quickly, recover from this error by opening an elevated command prompt and running gpupdate /force which forces a group policy refresh
Before we wrap-up, I would like to re-state one of Microsoft's long standing 'best practices' with respect to Hyper-V servers, and that is, the only Roles or Services that should ever be installed on a Hyper-V server is the Hyper-V Role and only those additional Roles or Features that directly support virtualization. The classic example is Hyper-V Failover Clusters where the Hyper-V Role and the Failover Clustering Feature complement each other by providing highly available virtualized workloads, which are the foundation of Microsoft's Cloud Strategy. If this 'best practice' is followed, no user rights modifications that could impact virtualization services should be needed.
I hope this has been helpful.
Thanks, and come back again soon.
Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support
High Availability\Virtualization Team
Comments
- Anonymous
January 01, 2003
Hey guys...make sure to read the KB article mentioned in the blog as well. - Anonymous
October 31, 2012
Great article! Thanks ;) - Anonymous
November 01, 2012
Rather than tell everyone to block GP inheritance, wouldn't it make more sense to modify the GP to grant NT VIRTUAL MACHINEVirtual Machines the SeServiceLogonRight? You're essentially telling people to turn off a security feature because it's easier. - Anonymous
November 01, 2012
Steve, I would agree with you but the problem I'm having is that these are Microsoft Hyper-v servers in a cluster and I cannot add the "VIRTUAL MACHINEVirtual Machines" account to the GPO since it is only local to the Hyper-Vs...Tom - Anonymous
December 05, 2012
Be sure to also review the KB mentioned in the blog - support.microsoft.com/.../2779204 - Anonymous
June 05, 2013
On Windows 8, getting logon failure , any suggestions? - Anonymous
June 12, 2013
Has there been a permanent fix for this yet. I like a good work-around as the next guy but I need a permanent solution for a production cluster that doesn't involve segregating servers in AD. - Anonymous
August 09, 2013
Just posted how to add NT Virtual MachineVirtual Machines "Special Identity" to a GPO:vdinotes.com/.../hyper-v-2012-logon-failure-errors-and-fix - Anonymous
March 25, 2014
In the related KB article (http://support.microsoft.com/kb/2779204), Method 2 mentions adding a specific user account to "Log on as a service". An alternate solution that worked for me was to set "Log on as a service" back to undefined by unchecking "Define these policy settings". The user account that was there was not necessary. - Anonymous
August 28, 2014
Why can't Microsoft just use a user account that actually exists? Why do we need "NT VIRTUAL MACHINEVirtual Machines?" It would be way easier if we could just specify the credentials to use, such as a domain admin account. - Anonymous
August 29, 2014
The comment has been removed - Anonymous
September 04, 2014
This issue is a pain... Unbelievable that Microsoft can not provide a real solution for this and we still have to play with GPs or local Policies. Careless development like often coming from Microsoft.