Hyper-V: Troubleshoot Pending Snapshot Merges

Article
1/17/2024

This article is based on a guest blog post by Jeremy Hagan at: http://blogs.technet.com/b/tonyso/archive/2010/11/25/the-case-of-the-pending-vm-snapshot-merge.aspx#3371639. Please help improve it; is is the wiki way!

When a virtual machine stops responding, frequently the underlying cause is that the hard-disk has run out of space. If you are using fixed-size VHD files, (so that there should be nothing that should grow to fill up the disk), check to ensure that the machine is NOT actually running on a differencing disk (AVHD) that had been a snapshot in the past, which changes still waiting to merge into the parent VHD file.

When a snapshot is deleted, the virtual machine must be powered off, not merely rebooted, in order for the data in the differencing disk to merge into the parent VHD. The VM must remain powered off until the merge is complete. If the VM is booted back up again before the merge is completed, then the merge process stops until the machine is powered off again. The bigger the AVHD, the longer the merge takes. Although SCVMM and Hyper-V manager have an indication in the GUI that a snapshot exists, when a snapshot is deleted it is deleted from the GUI right away even though the snapshot hangs around until the merge is complete. So if your VM in this state there may be no visible indication of it.

Usually, this occurs because the VM had a deleted snapshot for an unknown time period, which grew until it filled the underlying disk. If you cannot free up disk space, the merge can not take place even if the VM was turned off.

Many we maintain a (250 GB or so) LUN on each Hyper-V cluster that is used for staging and emergencies. You can use System Center Virtual Machine Manager (SCVMM) to migrate the (power off) VM over the network to this "holding" or "temporary" LUN and allow the merge to complete and then move it back to its home. Problem solved.

To prevent this, do not disable free disk space monitoring on LUNs hosting VMs. Since the LUNs routinely run low on space then SCOM will be routinely raising Chicken Little alerts that many ignore.

Here's a better way: alert on a pending merge.

To do this, examine the difference between virtual machines in three different states:

one without a snapshot
one with a snapshot
one with a deleted snapshot

Fix the issue from Virtual server: under IDE Controller in Settings window, you can remove the VHD file and then attach it again, this will not affect the virtual machine.

Start by downloading the PowerShell Management Library for Hyper-V. You can use configuration/global_settings/disk_merge_pending = true.

Use a VBScript to create a temporary PowerShell script in the current directory. Because the VBScript is distributed by SCOM it will be on local disk,, allowing you to use RemoteSigned certificates and keep your servers safe.
Execute the PowerShell script by using the Windows Script Host Exec Method, which allows you to capture the output using the StdOut object of the Exec method
Return the state of the Hyper-V host (whether or not there was a pending merge) and the name of the VM with the pending merge to the monitor via the property bag API
Set the machine to a Warning state and raise an alert including the output of the script
Create a rule to run the script and a monitor to alert on the results. Jeremy modified the script and abandoned the property bag and just wrote the information to the local Application event log and the monitor would raise the alert based on that information. He configured the rule to run every 24 hours and for the alert to expire after 23 hours, so SCOM alerts him every day about the pending merge until resolves.

Of course you could customise this in your own implementation.

PowerShell

foreach ($file in Get-ChildItem "$Env:PROGRAMDATA\Microsoft\Windows\Hyper-V\Virtual Machines\.xml" {
    $file.CopyTo((Split-Path -parent $MyInvocation.MyCommand.Definition) + "\ + $file.Name) | out-null
    [xml]$ConfigFile = Get-Content ((Split-Path -parent $MyInvocation.MyCommand.Definition) + "\ + $file.Name)
    $VMName = $ConfigFile.Configuration.properties.name."#text"
    $MergePending = $ConfigFile.configuration.global_settings.disk_merge_pending."#text"
    if ($MergePending -eq "True") {Write-Host "$VMName has a pending merge."}
    del ((Split-Path -parent $MyInvocation.MyCommand.Definition) + "\ + $file.Name)
    Remove-Variable VMName
    Remove-Variable ConfigFile
    Remove-Variable MergePending
}

Copy the configuration files to a temporary location before parsing them just in case

VBScript with embedded PowerShell

Option Explicit

Dim objFSO : Set objFSO = CreateObject("Scripting.FileSystemObject")
Dim objShell : Set objShell = CreateObject("WScript.Shell")
Dim strParentPath : strParentPath = objFSO.GetParentFolderName(WScript.ScriptFullName)
Dim objPSFile : Set objPSFile = objFSO.CreateTextFile(strParentPath & "\PSTemp.ps1", True)

'Get the path to the PowerShell executable from the registry
Dim strPowerShell : strPowerShell = objShell.RegRead("HKLM\SOFTWARE\Microsoft\PowerShell\1\ShellIds\Microsoft.PowerShell\Path")
Dim strCommand : strCommand = Chr(34) & strPowerShell & Chr(34) & " -NoProfile -NoLogo -File " & Chr(34) & strParentPath & "\PSTemp.ps1" & Chr(34)
Dim objExec
Dim strOutput : strOutput = ""
Dim i

objPSFile.WriteLine "foreach ($file in Get-ChildItem " & Chr(34) & "$Env:PROGRAMDATA\Microsoft\Windows\Hyper-V\Virtual Machines\.xml" & Chr(34) & ") {"
objPSFile.WriteLine "    $file.CopyTo((Split-Path -parent $MyInvocation.MyCommand.Definition) + " & Chr(34) & "\ & Chr(34) & " + $file.Name) | out-null"
objPSFile.WriteLine "    [xml]$ConfigFile = Get-Content ((Split-Path -parent $MyInvocation.MyCommand.Definition) + " & Chr(34) & "\ & Chr(34) & " + $file.Name)"
objPSFile.WriteLine "    $VMName = $ConfigFile.Configuration.properties.name." & Chr(34) & "#text" & Chr(34)
objPSFile.WriteLine "    $MergePending = $ConfigFile.configuration.global_settings.disk_merge_pending." & Chr(34) & "#text" & Chr(34)
objPSFile.WriteLine "    if ($MergePending -eq " & Chr(34) & "True" & Chr(34) & ") {Write-Host " & Chr(34) & "$VMName has a pending merge." & Chr(34) & "}"
objPSFile.WriteLine "    del ((Split-Path -parent $MyInvocation.MyCommand.Definition) + " & Chr(34) & "\ & Chr(34) & " + $file.Name)"
objPSFile.WriteLine "    Remove-Variable VMName"
objPSFile.WriteLine "    Remove-Variable ConfigFile"
objPSFile.WriteLine "    Remove-Variable MergePending"
objPSFile.WriteLine "}"
objPSFile.Close

Set objExec = objShell.Exec(strCommand)
objExec.StdIn.Close

'sanity check
Do While objExec.Status = 0
    WScript.Sleep 1000
    i = i + 1
    If i > 55 Then
        i = 0
        objExec.Terminate
    End If
Loop

Do While True
    If objExec.StdOut.AtEndOfStream Then
        Exit Do
    Else
        strOutput = strOutput & objExec.StdOut.Read(1)
    End If
Loop

If InStr(strOutput, "pending") Then objShell.LogEvent 2, strOutput

objFSO.DeleteFile strParentPath & "\PSTemp.ps1", True

Set objFSO = Nothing
Set objExec = Nothing
Set objPSFile = Nothing
Set objShell = Nothing