Best practices to upgrade a VM Scale Set between OS versions?

Question

If I have a Scale Set with a set of stateful VMs using local OS disk for data, what would be the best practices to upgrade the OS from eg Ubuntu 20 to Ubuntu 22? Is it possible to do it without data loss? Auto upgrade only supports minor versions.

Answer 1

Best Practices for OS Upgrade

A. If You Can Tolerate a Rebuild (Preferred Approach)

Back Up Critical Data

If necessary, move stateful data to attached managed disks, a remote file system (e.g., Azure Files, NFS), or a database before proceeding.

Create a Custom Image with Ubuntu 22

Deploy a new Ubuntu 22 VM.

Install required dependencies, software, and configurations.

Capture a Managed Image or create an Azure Compute Gallery image version.

Update the VMSS Model

Update the scale set to use the new image:

sh

Copy

Edit

az vmss update \

--resource-group <resource-group> \

--name <vmss-name> \

--set virtualMachineProfile.storageProfile.imageReference.id=<image-id>

Rolling Upgrade

Perform an orchestrated rolling upgrade:

sh

Copy

Edit

az vmss rolling-upgrade start --resource-group <resource-group> --name <vmss-name>

Use --max-unhealthy-instance-percent to control risk.

Test and Validate

Monitor logs and application behavior.

Adjust if needed before updating all instances.

B. If You Need to Preserve the OS Disk (Riskier Approach)

You can attempt an in-place upgrade, but it's not recommended due to potential incompatibilities and downtime.

Create a Backup Snapshot (If Possible)

If the OS disk is managed, create a snapshot.

For ephemeral disks, ensure external backups are in place.

Run the Ubuntu Release Upgrade

SSH into each VM and execute:

sh

Copy

Edit

sudo apt update && sudo apt upgrade -y

sudo do-release-upgrade -d

Follow prompts and restart.

Handle Failures Gracefully

If a VM fails to boot, you might need to manually recover or rebuild it from a working image.

Automate via Custom Script Extensions.

I would go for the safest method and make sure to have backup image