Best practices to upgrade a VM Scale Set between OS versions?

Question

If I have a Scale Set with a set of stateful VMs using local OS disk for data, what would be the best practices to upgrade the OS from eg Ubuntu 20 to Ubuntu 22? Is it possible to do it without data loss? Auto upgrade only supports minor versions.

Answer 1

Best Practices for OS Upgrade

A. If You Can Tolerate a Rebuild (Preferred Approach)

Back Up Critical Data

If necessary, move stateful data to attached managed disks, a remote file system (e.g., Azure Files, NFS), or a database before proceeding.

Create a Custom Image with Ubuntu 22

Deploy a new Ubuntu 22 VM.

Install required dependencies, software, and configurations.

Capture a Managed Image or create an Azure Compute Gallery image version.

Update the VMSS Model

Update the scale set to use the new image:

sh

Copy

Edit

az vmss update \

--resource-group <resource-group> \

--name <vmss-name> \

--set virtualMachineProfile.storageProfile.imageReference.id=<image-id>

Rolling Upgrade

Perform an orchestrated rolling upgrade:

sh

Copy

Edit

az vmss rolling-upgrade start --resource-group <resource-group> --name <vmss-name>

Use --max-unhealthy-instance-percent to control risk.

Test and Validate

Monitor logs and application behavior.

Adjust if needed before updating all instances.

B. If You Need to Preserve the OS Disk (Riskier Approach)

You can attempt an in-place upgrade, but it's not recommended due to potential incompatibilities and downtime.

Create a Backup Snapshot (If Possible)

If the OS disk is managed, create a snapshot.

For ephemeral disks, ensure external backups are in place.

Run the Ubuntu Release Upgrade

SSH into each VM and execute:

sh

Copy

Edit

sudo apt update && sudo apt upgrade -y

sudo do-release-upgrade -d

Follow prompts and restart.

Handle Failures Gracefully

If a VM fails to boot, you might need to manually recover or rebuild it from a working image.

Automate via Custom Script Extensions.

I would go for the safest method and make sure to have backup image

Answer 2

Hi Florian-6559,

Yes, is it possible to use canonical 22.04 base image directly in the upgrade and you can then apply your user-data script to configure your system post-start.

you can provide a custom script or user-data to execute upon startup.

Use Azure Backup or a custom backup solution .

local OS disk to Azure Blob Storage or Azure Managed Disks for additional safety.

To update your scale set to use a different OS version, you need to set all the updated properties in a single call. In this example, we are changing from Unbuntu Server 20.04 to 22.04.

az vmss update \ --resource-group myResourceGroup \ --name myScaleSet \ --set virtualMachineProfile.storageProfile.imageReference.offer=0001-com-ubuntu-server-jammy \ --set virtualMachineProfile.storageProfile.imageReference.publisher=Canonical \ --set virtualMachineProfile.storageProfile.imageReference.sku=22_04-lts-gen2 \ --set virtualMachineProfile.storageProfile.imageReference.version=latest

Please refer to this link for more information on Set to OS image upgrades:

https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-upgrade-scale-set#the-scale-set-model

https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-automatic-upgrade#manually-trigger-os-image-upgrades

If you found it helpful, could you kindly click the “Accept Answer and upvote” on the post.

If you have any further queries, please let us know we are glad to help you.