Freigeben über

How Hyper-V Backup Got Better in 2012 R2 (but now requires a SCSI controller)

  • Artikel

Hyper-V has always provided the ability to backup all running virtual machines from the host operating system, without incurring any downtime for the virtual machines being backed up.  To do this it utilizes the fantastic VSS (Volume Shadow Copy Services) functionality in Windows.  When a user wants to backup a Hyper-V server, the following process is followed for Windows Server 2008, 2008 R2 and 2012:

  1. The backup application requests that Hyper-V backup all virtual machines
  2. Hyper-V goes through each virtual machine and sends a request to the Hyper-V Backup integration service inside of the virtual machine
  3. The Hyper-V Backup integration service inside of the virtual machine then asks the guest instance of VSS to create a data-consistent VSS snapshot inside of the virtual machine
  4. Once all the virtual machines have confirmed that they have taken a VSS snapshot, Hyper-V tells the host instance of VSS that it can take a host VSS snapshot
  5. Now, at this stage - some time has elapsed between steps 3 and 4.  This means that the virtual machines were continuing to write data to their virtual hard disks, and that the virtual hard disks host VSS snapshot are not actually data consistent.  To handle this, Hyper-V then:
    1. Locates the copies of the virtual hard disks that are stored in the newly created host side VSS snapshot
    2. Mounts each of these virtual hard disks in the host environment
    3. Uses VSS to revert each virtual hard disks state to the VSS snapshot that was taken in step 3
    4. Dismounts each virtual hard disk
    5. Informs VSS that it is complete - and that it can return the files in the host VSS snapshot to the backup application

This is all fine and good, but it does have its drawbacks.  Particularly, it involves mounting a copy of every virtual hard disk on the system in the host environment.  So, for a computer with 100 virtual machines - each with 2 virtual hard disks - that means 200 virtual hard disks are mounted and dismounted every time a backup is taken.  Needless to say, this can cause interesting problems (like this one, for example).

 

To make the system more reliable, especially at scale, we have made a change to this process.  In Windows 2012 R2 backing up Hyper-V now follows this process:

  1. The backup application requests that Hyper-V backup all virtual machines
  2. Hyper-V goes through each virtual machine and sends a request to the Hyper-V Backup integration service inside of the virtual machine
  3. The Hyper-V Backup integration service inside of the virtual machine then asks the guest instance of VSS to create a data-consistent VSS snapshot inside of the virtual machine
  4. Once all the virtual machines have confirmed that they have taken a VSS snapshot, Hyper-V tells the host instance of VSS that it can take a host VSS snapshot
  5. Now, at this stage - some time has elapsed between steps 3 and 4.  This means that the virtual machines were continuing to write data to their virtual hard disks, and that the virtual hard disks host VSS snapshot are not actually data consistent.  To handle this, Hyper-V then:
    1. Locates the copies of the virtual hard disks that are stored in the newly created host side VSS snapshot
    2. Connects each virtual hard disk copy to the still running virtual machine that it was copied from
    3. Sends a message to the Hyper-V Backup integration service inside the virtual machine that instructs it to revert the virtual hard disk copy to the VSS snapshot that was taken in step 3
    4. Disconnects each virtual hard disk
    5. Informs VSS that it is complete - and that it can return the files in the host VSS snapshot to the backup application

As you can see - the big change here is that the action of reverting the virtual hard disk to the data consistent VSS snapshot now takes place inside the virtual machine instead of in the host operating system.  This has many benefits, one of which being that it scales excellently!  It does, however, have one (minor) drawback.  In order for this method to work - we need to be able to hot add and remove virtual hard disks to and from the virtual machine.  And this is something that is only supported on our SCSI controller (not our IDE controller).

 

Fortunately, we have always created virtual machines with a SCSI controller connected (even if there were no disks attached).  If, however, you have manually removed all SCSI controllers from a virtual machine - Hyper-V backup will now fail on that virtual machine.

 

Cheers,
Ben

Comments

  • Anonymous
    March 09, 2015
    Thanks

  • Anonymous
    March 10, 2015
    better than 2008 R2??

  • Anonymous
    March 13, 2015
    Ben, I'd been wondering for 5 years exactly how backups are made of live VMs. And now I know. Thank you! :-) This explains why it takes so long to kick off the backup, why VHDs were mysteriously being added and removed from the host, and why backups are an unreliable affair if just 1 VM doesn't respond positively to requests from the host. The error messages logged aren't easy to get to the bottom of. One important factor is there must be sufficient disk space on the VHDs to store snapshots and also, I think, the host's disks. I haven't worked out how much space exactly is required, it would be useful if there was a rule or thumb. Incidentally, I was the guy who got in touch with you in early 2010 to report the backup bug where 2008 R2 kept 10,000's of registry entries of every single snapshot made that prevented booting up. That put our deployment back 3 months while we worked with Microsoft to identify the issue and then wait for a fix. It highlights the risk of putting a major deployment onto a new platform so we might migrate to 2012 R2 rather than vNext, as much as I love the latest technologies. Back to your blog post... are you saying it's less likely for backups to fail under 2012 R2 compared to 2008 R2? When you make reference to a SCSI controller do you mean the virtual one in Hyper-V and not a physical one? If the VM's boot drives use IDE but the VM's data drives use SCSI would that be okay in 2012 R2 or will the boot VHDs on IDE break this? Thanks!

  • Anonymous
    March 18, 2015
    Ben, Is it enough just add SCSI controller to the VM with IDE disks ? Could you please answer.

  • Anonymous
    March 19, 2015
    For the last two posters, I think a rereading will yield the answers. Yes, he's talking specifically about the virtual SCSI controller, and as he said, one just has to exist, not have drives attached. Other drives may be IDE.

  • Anonymous
    April 09, 2015
    Does this apply to 2012 R2 Update as well, because the VSS backup format changed again in that release? Because I tried to backup a Gen 1 no VM in 2012 R2 Update with the SCSI Controller removed and had no problems, even with an IDE disk.  Also tried with a Gen 2 VM with no SCSI controller in 2012 R2 Update and had no problems backing it up.

  • Anonymous
    July 20, 2015
    Michael , Gen2 VM only supports SCSI so how did you tried backing up?

  • Anonymous
    June 12, 2017
    Awesome Read!Now as in server 2012 R2 we are not using Guest VM snapshot method, rather using checkpoints, so that we get clean copy of VHDX /VHD. However my ask is what is the use of autorecovery.avhdx in this context.