Orchestrated offline VM Patching using Service Management Automation
My Name is Thomas Roettinger and I am a Program Manager in the Windows Server System Center Customer Architecture & Technology Team working in the fabric management space together with various teams.
In this cycle of customer validation I got asked by various people what happened to the Virtual Machine Servicing Tool (VMST). The last version of VMST was supposed to work with Virtual Machine Manager 2012.
Virtual Machine Servicing Tool (VMST) 2012
In those conversations I was listening very carefully about the value VMST gave to them and what they are missing today. The answer was always the same. The problem we need to solve is to ensure that whenever a tenant deploys a virtual machine it already includes the latest Microsoft Security Updates. This helps to guarantee the best possible experience for our customers allowing virtual machines to be ready available and not requiring patching upon first boot.
So the goal is to ensure images stored in the Virtual Machine Manager library are updated as soon as Microsoft Security Updates become available on Patch Tuesday. In this post I’ll present you a way to achieve that goal, while leveraging the entire CLOUD OS stack including Windows Server 2012 R2, System Center 2012 R2 and Windows Azure Pack.
The solution itself is provided by a SMA (Service Management Automation) Runbook. If you have not looked into SMA I highly recommend you to do so. My peers have already done great blog posts for Automation that you want to check out here.
High Level Overview
1. Get all Images stored in the Library
2. Mount Image on VMM Server
3. Get available Updates from WSUS
4. Check if Update is already installed and applicable
5. Apply Updates offline that meeting condition to the mounted Images
6. Commit Changes to mounted Image
7. Do Step 2-6 for all available Images
Runbook Example
In this section you get the sample code and I share considerations when using the sample code. You can also download this Runbook example and directly import it in your environment. I used the SMA Runbook Toolkit to do the export and you will need it for the import.
Orchestrated Offline VM Patching Runbook
Image Requirements
- An Operating System must be configured for the Virtual Hard Disks
- The Virtualization Platform must be set to “Microsoft Hyper-V”
The current version of Deployment Image Servicing and Management tool “DISM” and the related PowerShell commands do not allow to scan an image for required updates. So the script applies every update available on the WSUS server what is for sure not the best way but the only way today!
Let’s look at some of the main commands used in more detail
Get-WindowsPackage, evaluates if an update is applicable to the image. As you may notice in the code example below this functionality is only available for cab packages but not for msu packages. The reference for this command can be found here. The command does also not check for update dependencies meaning you may see updates failing to install because of a dependency.
Add-WindowsPackage, adds a single .cab or .msu file to the mounted windows image. The reference for this command can be found here. The important take away is that only Windows Server 2008 SP1/Vista SP1 and newer are supported to update. For more details you can check out this technet article.
Deployment Image Servicing and Management Technical Reference
Basic Code Example
| 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 | <# .SYNOPSIS Orchestrated offline VM Patching using Servce Management Automation (VM-Offline-Patching.ps1) Written by Thomas Roettinger Windows Server System Center, Customer and Technologies Team (WSSC CAT) Microsoft Corporation - 12-06-2013 .Requirements This script requires variables to be set in Service Management Automation. The detailed steps can be found in the corresponding blog post https://blogs.technet.com/b/privatecloud/ #> workflow VM-Offline-Patching { # Connection to access VMM server. $VmmConnection = Get-AutomationConnection -Name 'VmmConnection' $VmmServerName = $VmmConnection.ComputerName # Create a PSCredential from the 'Username' and 'Password' fields within # 'VmmConnection' because this is the form of authentication that an # inlinescript accepts. $SecurePassword = ConvertTo-SecureString -AsPlainText -String $VmmConnection.Password -Force $VmmCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $VmmConnection.Username, $SecurePassword #Connection to access WSUS Server $WSUS = Get-AutomationVariable -Name 'WSUSSERVER' inlinescript { # Import VMM module. Import-Module virtualmachinemanager # Connect to VMM server. Get-SCVMMServer -ComputerName $Using:VmmServerName #Get all Virtual HardDisks stored in the library, check if virtualization platform is Hyper-V, Check for Operating System $VHDList = Get-SCVirtualHardDisk | where-object {$_.VirtualizationPlatform -eq "HyperV"}|where-object {$_.OperatingSystem -ne "none"} #Get all available Update at WSUS $Updatelistcab = get-childitem -Path "\\$using:wsus\wsuscontent" -include *.cab -recurse -File $Updatelistmsu = get-childitem -Path "\\$using:wsus\wsuscontent" -include *.msu –recurse -File #Mount and Patch each Virtual HardDisk, check Update if applicable before applying Foreach ($VHD in $VHDList) { $VHDPath=$VHD.SharePath Mount-WindowsImage -ImagePath "$VHDPath" -Path "c:\ovp" -Index 1 Foreach ($Updatecab in $Updatelistcab) { $UpdateReady=get-windowspackage -PackagePath $Updatecab -Path "c:\ovp" If ($UpdateReady.PackageState -eq "installed") {Write-Output $UpdateReady.PackageName "is already installed"} elseif ($updateReady.Applicable -eq "true") {Add-WindowsPackage -PackagePath $Updatecab.Directory -Path "c:\ovp"} } Foreach ($Updatemsu in $Updatelistmsu) { add-windowspackage -PackagePath $Updatemsu.Directory -Path "c:\ovp" } Dismount-WindowsImage -Path "c:\ovp" -save } }-PSComputerName $VmmServerName -PSCredential $VmmCredential } |
Considerations when using the basic code sample.
1. The sample code is targeting all images VHD or VHDX in the library that are Hyper-V compatible and contain an Operating System. But you can modify the command to only target specific images by name, filter by OS or other parameters see here.
Here are some example to extend the basic code sample
| 001 002 003 | #Get a specific Virtual Hard Disk in the library by Name $VHDList = Get-SCVirtualHardDisk -VMMServer $Using:VmmServerName -Name test.vhdx |
| 001 002 | #Get all Virtual HardDisks VHD & VHDX in the library with Virtualization Platform Hyper-V running Windows Server 2012 R2 $VHDList = Get-SCVirtualHardDisk -VMMServer $Using:VmmServerName|where-object {$_.VirtualizationPlatform -eq "HyperV"}|where-object {$_.OperatingSystem -eq "Windows Server 2012 R2 Datacenter"} |
Runbook Setup
1. Ensure that all images in the Virtual Machine Manager library that you want to patch have Operating System and Virtualization Platform specified. The Virtualization Platform must be “Microsoft Hyper-V”
2. Start with creating a folder on your Virtual Machine Manager Server and call it “c:\ovp” . This folder is used to mount your images that you have stored in your VMM library.
3. Inside the script we do double security hops. The workflow runs on the SMA machine but gets executed on the VMM machine. This part also calls the WSUS Server to get the available updates and this is where we do the double hop. The easiest way to address this is by using Active Directory delegation and enable delegation for your VMM Server.
If you want to switch to use credssp in your environment I recommend you to review the blog post below
Enable PowerShell “Second-Hop” Functionality with CredSSP
4. Open your Windows Azure Pack Admin Portal and go to the Automation resource provider on the left hand side. This blog assumes you have already installed and registered Service Management Automation in the WAP Admin Portal. If not you can find detailed information here: https://technet.microsoft.com/en-us/library/dn469260.aspx
5. Next step is to configure the VMM connection and the WSUS variable. You do this by clicking “Assets” in the top menu. Buttons do appear at the end of the site where you click “Add Setting”.
6. Start with adding the VMM connection. Select “Add Connection” and enter ”VmmConnection” as name, then the appropriate credentials and the VMM machine name.
7. Repeat the steps by clicking “Add Setting” again but this time you select “Add Variable”. As Variable Type select “String” and as Name enter “WSUSSERVER”. In the next screen you enter the server name of you WSUS machine.
8. Now it is time to create the Runbook that does the patching job. Click the plus and select “Quick Create”. Give the Runbook a name for example “VM-Offline-Patching” you may also consider assigning a TAG to identify your custom Runbooks.
9. Switch to your newly created Runbook, select AUTHOR and create a new draft. This allows you to copy & paste the example code into you draft Runbook.
10. Save the script and use the test button to give it a try. For you first test run you may only target one specific image in your VMM library. Check the Runbook Example section in the post for how to achieve that. If the Runbook Test was successful you are ready to publish it don’t forget to change if back to patch all your images. If the Test was not successful you need to walk through the Troubleshooting section in this post.
11. You can now schedule or execute the Runbook manually. The Job should successfully finish like in the example below.
12. Execution of the Runbook may take while so let it run on its course depending on number of images and number of available updates. When the Runbook execution is finished your last step is to deploy a new Virtual Machine. The fastest way to check if the newly created virtual machine contains all available security updates is to simply run Windows Updates and check how many updates you get offered. Needless to say it should be close to zero.
Note: The following command does not list the installed hotfixes since the security updates are incorporated offline into the image. Keep that in mind just in case you thought this would be another smart way to identify if updates have been applied.
| 001 | get-hotfix |
Troubleshooting the Runbook
How to troubleshoot the Runbook when you see errors
1. If you get an error in the Output screen like this:
The converted JSON string is in bad format. (The converted JSON string is in bad format. (The converted JSON string is in bad format.))
Then you ran into a permission problem, remember the double security hop we do. Go and check the AD delegation settings.
2. If you see errors like this one
Error: Mount-WindowsImage : The specified VHD does not have a system volume.
Then you forgot to assign an Operating System to the image and the Runbook mounted a blank virtual hard disk that is in the library per default
3. You may also see those errors if an update was already applied to an image so you can ignore it
Error: Add-WindowsPackage : There is no valid package in the path:\\wsus.contoso.com\wsuscontent\91
4. If the Runbook completes after a minute or two without any issue then you forgot to assign the Virtualization Platform Microsoft Hyper-V to your library Images. The result is that the Runbook did not found any images.
Finally I hope this Runbook example is useful for you and I am interested in hearing your feedback. Let me know how long it runs, how many images you have in the library and how many different Windows Versions your WSUS offers updates for. Since those are all important aspects when going for scale. So please hop over to the Forum and share your experience!
Enjoy your complete patched VM after deployment!
Thomas
Comments
Anonymous
December 07, 2013
Can the patching somehow be done by using SCCM not by WSUS?Anonymous
December 09, 2013
The reason using WSUS is that in the CLOUD OS stack you have VMM that already manages WSUS to patch the fabric. Leveraging the same WSUS server ensures you have a central console to approve security updates for the fabric and Tenant images but also drives consistency in applied security updates.Anonymous
December 20, 2013
Pingback from Windows Azure Pack Blog Post Overview on Building Clouds & TechNet - Building Clouds Blog - Site Home - TechNet BlogsAnonymous
January 06, 2014
Pingback from Windows Azure Pack Blog Post Overview on Building Clouds & TechNetAnonymous
January 06, 2014
Pingback from Automation ??? Orchestrator Back to Basics ??? Use Cases Spotlight (2 of 5)Anonymous
May 21, 2014
This is post #2 out of 5 in the “ Orchestrator – Back to Basics – Use Cases ”Anonymous
August 23, 2014
I work with SCVMM (System Center Virtual Machine Manager) frequently in many different contexts. I evenAnonymous
August 23, 2014
I work with SCVMM (System Center Virtual Machine Manager) frequently in many different contexts. I evenAnonymous
August 27, 2014
Thank you dear Thomas,
If I have only System Center Orchestrator without Windows Azure Pack and SMA, can use the same Runbook?
Best regards,