Share via

Hyper-V 2008/2008 R2: Gotchas

  • Article

Note: This article is based on Hyper-V 2.0 and might not apply to Hyper-V 3.0 (Server 2012)

TechEd Demo Edit
These are the top "gotchas" or "known issues" that folks seem to have trouble with in Hyper-V. "Gotchas" are sometimes also called "lessons learned" and "best practices." It is a best practice to avoid a known "gotcha". Please add any you have encountered.
Applies to:

  • All versions of Hyper-V

**
TIP**: Subscribe to the RSS feed for this wiki page to get auto-notification when it is updated! To see when this topic was last updated, and with what information, click the "History" tab.

Live Migration Fails if Disk Space Low

When running a live migration using SCVMM 2008, live migration failed with Error (10698) Virtual machine [guest] could not be live migrated to virtual machine host [host] using this cluster configuration.  (Unspecified error (0x80004005)) 

When trying the migration from the Failover Cluster Manager, the job fails with the following;

Event 1069 Cluster resource '%1' in clustered service or application '%2' failed. 

Event 1205 The Cluster service failed to bring clustered service or application '%1' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.

I found that the clustered disk holding the server VHD and resource files was very low on space (2gb free in my case). Also, note that the server I'm migrating has a dynamically expandable drive which could be relevant

I then tried to live migrate the storage using SCVMM of one of the machines and the job failed with Error (12700) VMM cannot complete the Hyper-V operation on the [host] server because of the error: [guest] failed to save RAM contents during a snapshot operation: The I/O operation has been aborted because of either a thread exit or an application request. (0x800703E3). (Virtual machine ID xxx).

Fortunately, I managed to free up disk space by deleting an old test guest VM after which I could freely live migrate once again.

Alternatives would have been to add more disk space or manually move the disk after powering down the box.

Not enough disk space

When setting up your virtual machines, be aware that the virtual machine may try to cache a running copy of memory (using .bin files) to the HDD its configuration info is hosted on. Leave yourself enough HDD space for as much RAM as you wish to assign. Consider putting the main VM folder on a dedicated drive; apart from your other VHDs, if not using a RAID 5/6 array.

Using Hyper-V snapshots

  1. New! If you are using Windows Server 2012  many of the issues noted here are not going to be an issue for you! For details see Ben Armstrong's blog Live snapshot merging! [Windows Server “8”] 
    1. If you are *not* running Windows Server 2012 read on.
  2. Watch Microsoft Hyper-V Program Manager Ben Armstrong discuss common Hyper-V snapshots in this video.
  3. Snapshots are not designed to serve the same function as backups. They are moments in time that you can return to. They still need to be backed up to recover from host or storage failure.
  4. Read the Hyper-V Snapshot FAQ.
  5. Remember that if you revert a domain-joined VM to a snapshot that has been offline for a longer time period than the machine account password reset interval in your environment, your VM will not be able to logon to the domain.
    1. See the Active Domain Controllers in VMs section of this document.
  6. Snapshots fail to merge with error 0x80070070 - This occurs because of low disk space. The resolution is to free up disk space to allow the merge to complete or move the .VHD and .AVHD file to a volume with sufficient disk space and manually merge the snapshots.
    1. One method to work around this is with Hyper-V R2 and newer is to take a snapshot and export that snapshot to a new virtual machine. This will result in a new VM with a single VHD. Import this Exported VM, verify it works and then delete the original and its virtual disks.
  7. Virtual machines show a state of "Paused-Critical" - This is caused by a lack of free disk space on the volume hosting the .vhd or .avhd files. The resolution is to free up disk space on the volume hosting the .vhd or .avhd files, as mentioned in the Hyper-V Snapshot FAQ.
    1. This commonly happens because snapshots quickly increase the amount of storage that is consumed.   
  8. Snapshots are lost when the parent VHD is expanded - There is a warning about this in the Edit Disk Wizard, but some folks miss it. This is also discussed in the

Hyper-V Snapshot FAQ - another reason to read it before using snapshots in production.

http://blogs.msdn.com/b/virtual_pc_guy/archive/2009/04/20/why-does-it-take-so-long-to-delete-a-virtual-machine-with-snapshots-hyper-v.aspx

Backing up and restoring virtual machines

Hyper-V does not have a built-in method for backing up and restoring individual virtual machines. This requires the use of Windows Server Backup or an external product such as Data Protection Manager, or many of the third-party backup products.

  1. Snapshots are not backups. Please see the snapshot topic.
  2. Avoid simultaneous backup operations from the guest and from the host. This usually happens by accident through scheduled backups. The result is heavy disk IO that affects both backup jobs.

On Windows Server 2008 R2, virtual machines lose network connectivity under a heavy load

This issue is caused by a race condition in the hypervisor. For Windows Server 2008 R2 systems that have Service Pack 1 installed, install hotfix KB2263829. For Windows Server 2008 R2 systems that do not have Service Pack 1 installed, install hotfix KB974909. 

Other Hyper-V networking related hotfixes to be aware of for non-SP1 systems:

Hyper-V Hotfix for "0x00000101 - CLOCK_WATCHDOG_TIMEOUT" on Nehalem systems

This issue is actually caused by a bug (or in Intel terms “erratum”) that causes these processors to generate spurious interrupts.  You can read about this in the latest Intel documentation for this process series: http://www.intel.com/assets/pdf/specupdate/321324.pdf. To fix this: Install hotfix KB975530 or install Service Pack 1 for Windows Server 2008 R2 (KB976932). 

On a Windows Server 2008 or 2008 R2 system that has the Hyper-V role installed, the startup time increases after backing up Hyper-V virtual machines

This is caused by registry bloating in the management operating system due to registry keys not being deleted after the Hyper-V virtual machines are backed up. To fix this follow the instructions in Microsoft Knowledge Base Article KB982210, "The startup time increases every time after you back up the data on all the disks of the computer in Windows Server 2008 R2 if the computer runs some virtual machines" to use the devnodeclean utility to remove the orphaned registry entries.

VM unable to access the network after it has been configured to use a VLAN ID

This is generally caused by a mismatch between the VLAN ID used by the virtual machine and the VLAD ID configured on the physical network switch. The resolution is to configure the virtual machine to use the VLAN as covered in the Hyper-V Planning and Deployment guide.

NIC driver issues using NIC Teaming on Broadcom NICs on Windows Server 2008 R2

BIOS errors enabling Hyper-V

Some computer BIOS changes require that the machine is physically powered off, and then back on before the change will take effect. Just "restarting" the server is not sufficient. For more information see:

Errors enabling Hyper-V

Most instructions for enabling Hyper-V assume you have not done anything else to the machine beforehand. If you are re-purposing an existing machine, you must take special steps if you are using BitLocker Encryption or Antivirus on that machine, since both will generate errors when you enable Hyper-V. It is best to turn off both BitLocker and AV before enabling the Hyper-V role. See also:

Creating or starting a virtual machine fails with the error:

'General access denied error' (0x80070005)

This is caused by incorrect permissions on the .vhd, .avhd, or .xml file. The resolution is to perform the steps documented in KB2249906.

VM Export/Import permissions issues

To address issues with exporting and importing Hyper-V virtual machines, see the following:

  1. Blog posting that discusses VM Export/Import permissions issues.
  2. Some customers have run into problems when trying to export a VM to a file share and get an access denied error. This is because both the machine account and the user account must have access to the share (often times only the user account has access). This is explained and demonstrated starting at timecode 2:43 into this video.

NIC listed as an unknown device in device manager

The cause of this is that the correct Integration Services (formerly called Integration Components, or ICs) are not installed. To fix this, install the correct integration services on the VM according to these instructions in the TechNet library.

Trying to run Hyper-V on a laptop

Running Hyper-V on a laptop is unsupported. For further information on how to make it work, see http://social.technet.microsoft.com/wiki/contents/articles/hyper-v-how-to-run-hyper-v-on-a-laptop.aspx

Trying to run Hyper-V on an AMD E-350 motherboard

AMD-E350 motherboards provide a very budget cost-efficient way to get a test lab with Hyper-V installed. Ensure that the SATA controller is in AHCI mode; otherwise, the OS will hang on bootup once the Hyper-V role is installed.

Trying to PXE-boot a VM using legacy network adapter

Hyper-V VM's are only able to PXE-boot if no bootable disk has been detected. If a disk has been detected that is marked "active", the legacy network adapter will not even attempt to load the PXE-boot ROM and skip through to boot from CD/IDE disk. The priority of the boot-order does not matter in this case.
If you try to use the same VM for each time you need to test out an OS installation (creating reference images, testing final image deployment and such), it's best to create a snapshot the moment the VM has been created. This allows you to revert back to a clean disk state very quickly allowing you to PXE-boot again, without needing to delete&create a new vhd.

Disk I/O trouble

Be sure to plan proactively for the performance you require. One guideline is to be generous with RAM - more memory means less disk access. In general, a fixed virtual hard drive is faster than a dynamic virtual drive. Ensure that correct integration services are installed in the VM.

  1. For more detailed information on planning and performance, see the links in the Hyper-V Survival Guide.
  2. Read Measuring Performance on Hyper-V
  3. Read Tony Voelm's blog post Hyper-V Performance FAQ.

Running un-needed software, such as anti-virus on the parent partition (also called virtualization server and management partition)

If the only thing you are running on the virtualization server is virtual machines, why do you need to run anti-virus or other software that you normally run on a Windows Server 2008 R2 server? Review this short video showing what happens when you enable the Hyper-V role, and change your software installation so that all the VMs get the correct software installed as required by your IT policy. Review the best practice recommendations at Planning for Hyper-V Security and evaluate if they fit your environment. If your IT policy requires AV in the management operations system, see the list of Hyper-V AV exclusions in the primary AV exclusion KB for Enterprise customers (KB822158).

When you attempt to create or start a virtual machine, you receive one of the following errors: 

  1. The requested operation cannot be performed on a file with a user-mapped section open. ( 0x800704C8 )

  2. Failed to Power On with Error 'The specified network resource or device is no longer available.' (0x80070037).

  3. The I/O operation has been aborted because of either a thread exit or an application request. (0x800703E3)

These issues can be caused by antivirus software that is installed in the parent partition and the real-time scanning component is configured to monitor the Hyper-V virtual machine files. The resolution is to perform the steps documented in Microsoft Knowledge Base article KB961804, Virtual machines are missing in the Hyper-V Manager Console or when you create or start a virtual machine, you receive one of the following error codes: "0x800704C8", "0x80070037" or "0x800703E3".

Slow system logon, sluggish keyboard, and mouse

Task Manager shows all CPU cores with high utilization. This problem seems to be related to the version of the network card driver included in Windows 7 or Windows Server 2008 R2 and affects only certain systems, as described in this article from Intel, which also describes the workaround: http://www.intel.com/support/motherboards/server/sb/CS-031086.htm. The workarounds are to:

  1. Download the latest network card driver package from http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=18388. or
  2. Attach a network cable on both of the network ports, it can be attached to a switch or just back to back. This needs to be done prior to loading the driver, or
  3. Utilize the Intel® Deployment Assistant to install the operating system, this would enable the operating system to use the latest network card driver from http://www.intel.com/.

Misconfiguration

For example, on the free Microsoft Hyper-V Server 2008 R2, which is command-line only, remember that the command to turn on BitLocker is case sensitive:

ocsetup bitlocker à fail

ocsetup BitLocker  à works

In fact, OCSetup.exe (as well as ServerManagerCmd.exe) is considered deprecated in Windows Server 2008 R2 and Hyper-V Server 2008 R2. You're supposed to use new Server Manager PowerShell cmd-lets. The good thing about these cmdlets is that they are not case sensitive (as everything in PowerShell).

But there may be special considerations when Server Manager PowerShell cmdlets are not enough. Consider the following scenario. You install Microsoft System Center Data Protection Manager (SCDPM) in Virtual Machine. You need to install the Hyper-V Server Role to enable Item-level recovery (ILR) from backed up virtual machines. But the installation of Hyper-V role normally requires the presence of Hardware-Assisted Virtualization (HAV). This feature is not available in Virtual Machines. So you need to bypass this check. In order to do it, you have to use another command-line tool that is also new to Windows 7 and Windows Server 2008 R2 and called “Deployment Image Servicing and Management (DISM)”. And this tool is a case-sensitive one. So you should type the following command exactly as shown.

dism.exe /online /enable-feature /featurename:Microsoft-Hyper-V

Other common misconfiguration issues include:

  • Not planning for the lack of UI in the free version of Microsoft Hyper-V Server. There is not much difference in resources required between Enterprise/Datacenter with Hyper-V and the free version - Microsoft Hyper-V server. However, if you do not already have experience managing Windows Server Core installations, the lack of UI may cause some confusion at first.

  • Not planning for high availability for VMs, and not changing configurations for Live Migration. For example, when a VM in a cluster in hyper-v switches to another node, you may see a configuration error message indicating "virtual network missing". To resolve these issues you must select the Refresh Virtual Machine Configuration option in the Failover Cluster Management MMS when a VM configuration changes.

For more information and best practices for the design and operation of highly available VMs, see the following documentation:

  1. Hyper-V: Using Hyper-V and Failover Clustering
  2. Hyper-V: Live Migration Network Configuration Guide
  3. Failover Clustering for Windows Server 2008 Hyper-V with File Server Storage
  4. Hyper-V: Using Live Migration with Cluster Shared Volumes in Windows Server 2008 R2
  5. Windows Server 2008 R2 Hyper-V Performance Analysis: How You Can Get the Most out of Hyper-V

Active Directory Domain Controllers in VMs

Never save state or pause a DC. Always shut down virtual Domain Controllers properly to avoid replication errors. Pausing a Domain Controller for a long period of time before resuming it can interfere with timely replication. If you do pause the domain controller for a long time, replication may stop and cause lingering objects.

how lingering objects come to life and how the Service pack level affects the tombstone lifetime

Backup and restore Domain Controllers the right way

Only use the standard way to backup and restore Active Directory Domain Controller, since the default checks of Active Directory consistency will only be performed when the Domain Controller is aware of a restore.

Using imaging solutions like Handy Backup, Symantec Ghost, Acronis TrueImage seem like smarter backup and restore solutions, but they are imaging solutions (in contrast to proper backup and restore solutions.) Restoring an image of a Domain Controller may cause an Update Sequence Numbers (USN) rollback situation to occur, which may lead to replication problems and other undesired effects. More information here.

Like shutting down a virtual Domain Controller and start up a version of the VHD file from a while back, you want to avoid improperly using imaging solutions.

To avoid USN Rollback and Invocation ID problems but still using imaging solutions or even copying back old versions of a VHD file (big No-no), be sure to check out this post on backing up and restoring Active Directory in this unsupported manner.

Beware of pending snapshot merges

When you delete a snapshot on a running virtual machine it disappears from the GUI immediately, but the snapshot is not merged into the parent VHD until the VM is powered off and remains powered off until the merge is complete.  This can cause issues if the volume runs out of space as the AVHD grows and grows. Eventually, the VM will pause and you will not have sufficient space to merge the snapshot even if the machine is powered off.
The solution is to monitor for pending deleted merges. This can be done by monitoring this entry in the config file of each VM:

configuration/global_settings/disk_merge_pending = true

Importing a VM into a Hyper-V Failover Cluster

When importing a VM into a Hyper-V Failover Cluster you need to make sure that the import is taking place on the machine/node that is the owner of the Cluster Shared Storage. If you try to do the import on any other machine/node in the cluster, you will get a general permission denied error.

See this article on a SCOM monitoring solution for this issue.

Do you have more gotchas to add?

Click the Edit button above this article and add your issue - WITH the fix or workaround, please. Thanks in advance.