Release notes for AKS engine on Azure Stack Hub
Applies to version v0.80.2 of the AKS engine.
This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Upgrade considerations
- Are you using the correct marketplace items, AKS Base Ubuntu 20.04 Image Distro or AKS Base Windows Server for your version of the AKS engine? You can find the versions in the section Download new images and AKS engine.
- Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
- Are you using a reliable machine to run the AKS engine and from which you're performing upgrade operations?
- If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
- If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
- Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
- No system updates or scheduled tasks are planned.
- Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Use the upgrade command
You must use the aks-engine upgrade command as described in Upgrade a Kubernetes cluster on Azure Stack Hub.
Upgrade interruptions
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster can take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu image and AKS engine.
As explained in the documentation for AKS engine for Azure Stack Hub, deploying a Kubernetes cluster requires:
- The aks-engine binary (required).
- AKS Base Ubuntu 16.04-LTS Image Distro (deprecated - no longer use, change in API Model to use 20.04 instead).
- AKS Base Ubuntu 18.04-LTS Image Distro (deprecated - no longer use, change in API Model to use 20.04 instead).
- AKS Base Ubuntu 20.04-LTS Image Distro (required for Linux agents).
- AKS Base Windows Server Image (one of the following images is required for Windows agents):
- AKS Base Windows Server Image Containerd.
- AKS Base Windows Server Image Docker.
New versions of these images are available with this update:
Check the AKS engine and Azure Stack version mapping table for the required AKS base images.
Follow the instructions in Add the Azure Kubernetes Services (AKS) engine prerequisites to the Azure Stack Hub Marketplace.
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
Upgrading Kubernetes clusters created with the Ubuntu 16.04 distro
Starting with AKS Engine v0.63.0, the Ubuntu 16.04 distro is no longer a supported option, as the OS has reached its end-of-life. For AKS Engine v0.67.0 or later versions, the aks-engine upgrade automatically overwrites the unsupported aks-ubuntu-16.04 distro value with with aks-ubuntu-18.04. For AKS Engine v0.75.3 or later versions, if you're using Kubernetes v1.24 or above, the aks-engine-azurestack upgrade automatically overwrites the unsupported aks-ubuntu-16.04 distro value with aks-ubuntu-20.04.
Upgrading Kubernetes clusters created with the Ubuntu 18.04 distro
Starting with AKS Engine v0.75.3, the Ubuntu 18.04 distro is no longer a supported option, as the OS has reached its end-of-life. For AKS Engine v0.75.3 or later versions, the aks-engine-azurestack upgrade automatically overwrites the unsupported aks-ubuntu-18.04 distro value with aks-ubuntu-20.04.
Upgrading Kubernetes clusters created with docker container runtime
In Kubernetes v1.24, the dockershim component was removed from kubelet. As a result, the docker container runtime is no longer a supported option. See the Kubernetes v1.24 release notes for more information. For AKS Engine v0.75.3 or later versions, the aks-engine-azurestack upgrade automatically overwrites the unsupported docker containerRuntime value with containerd.
For AKS Engine release v0.75.3, clusters with Windows nodes on Kubernetes v1.23 can use the Windows base image with the Docker runtime. Clusters with Windows nodes on Kubernetes v1.24 can use the Windows base image with the Containerd runtime.
AKS engine and Azure Stack version mapping
| Azure Stack Hub version | AKS engine version |
|---|---|
| 1910 | 0.43.0, 0.43.1 |
| 2002 | 0.48.0, 0.51.0 |
| 2005 | 0.48.0, 0.51.0, 0.55.0, 0.55.4 |
| 2008 | 0.55.4, 0.60.1 |
| 2102 | 0.60.1, 0.63.0, 0.67.0, 0.67.3 |
| 2108 | 0.63.0, 0.67.0, 0.67.3, 0.70.0, 0.71.0, 0.73.0, 0.75.3, 0.76.0 |
| 2206 | 0.70.0, 0.71.0, 0.73.0, 0.75.3, 0.76.0, 0.77.0 |
| 2301 | 0.75.3, 0.76.0, 0.77.0, 0.78.0, 0.79.0 |
| 2306 | 0.78.0, 0.79.0, 0.80.2 |
| 2311 | 0.80.2* |
| 2406 | 0.80.2*, 0.81.1* |
| 2408 | 0.80.2*, 0.81.1* |
*Supported. See the AKS Engine Version Support policy for more information.
AKS engine and corresponding image mapping
You can find the supported Kubernetes versions for AKS Engine on Azure Stack Hub in the following table. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
Note
*Starting from Kubernetes v1.21, only the Cloud Provider for Azure is supported on Azure Stack Hub.
Note
** Starting from Kubernetes v1.24, only the containerd container runtime is supported. See the section Upgrading Kubernetes clusters created with docker container runtime for more information.
Update planning
The AKS engine upgrade command fully automates the upgrade process of your cluster, and handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
What's new with AKSe 0.76.0
- Added support for Kubernetes v1.24.11 and v1.25.7.
- You can find other features at the v0.76.0 GitHub page.
What's new with AKSe 0.75.3 and above
AKS Engine release v0.75.3, and all future AKS Engine releases on Azure Stack Hub, will be from the new aks-engine-azurestack repo. As such, all aks-engine commands should be replaced with aks-engine-azurestack. Commands to get the latest AKS Engine release on Azure Stack Hub have also changed. You can see the new commands in Create Linux client and Create Windows client. Create an issue in the new repo if you find any issues.
AKS Engine release v0.75.3 on Azure Stack Hub offers Ubuntu 20.04 LTS as its Linux base image. Starting with this release, Ubuntu 18.04 is no longer supported. See Upgrading Kubernetes clusters created with the Ubuntu 18.04 Distro for more information.
Starting from Kubernetes v1.24, only the containerd runtime is supported. See Upgrading Kubernetes clusters created with docker runtime for more information. For AKS Engine release v0.75.3, clusters with Windows nodes on Kubernetes v1.23 can use the Windows base image with Docker runtime. Clusters with Windows nodes on Kubernetes v1.24 can use the Windows base image with Containerd runtime.
You can find more features on the v0.75.3 GitHub page.
Instructions to use AKS engine 0.70.0 and above
Microsoft upgraded the Azure Cloud Provider in version 0.70.0. The Azure Cloud Provider is a core component shared between AKS Azure and AKS engine on Azure Stack Hub.
To use AKS engine 0.70.0 and above:
If you're attempting to create a new Kubernetes cluster for the first time: Use the sample API model provided for the appropriate version in the AKS engine and corresponding image mapping table.
If you're creating a new cluster, but want to use your existing API model: Modify your API model by following the Cloud Provider for Azure instructions. Failure to include the new setting results in a deployment error.
If you're using storage volumes: Make sure that you're using the AzureDiskCSI driver. Version 0.70.0 and above only support CSI drivers, not the legacy in-tree storage provider. To upgrade, follow the instructions in upgrade while using storage volumes.
Known issues
- Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
- Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond those previously listed. You can ignore them and use the version table above as the official guide for this update.
- AKS Engine v0.67.0 uses the wrong Windows image when deploying Windows clusters. Use v0.70.0 to resolve this issue.
Reference
The following list is the complete set of release notes for Azure and Azure Stack Hub combined:
- https://github.com/Azure/aks-engine/releases/tag/v0.64.0
- https://github.com/Azure/aks-engine/releases/tag/v0.65.0
- https://github.com/Azure/aks-engine/releases/tag/v0.65.1
- https://github.com/Azure/aks-engine/releases/tag/v0.66.0
- https://github.com/Azure/aks-engine/releases/tag/v0.66.1
- https://github.com/Azure/aks-engine/releases/tag/v0.67.0
- https://github.com/Azure/aks-engine/releases/tag/v0.70.0
- https://github.com/Azure/aks-engine/releases/tag/v0.71.0
- https://github.com/Azure/aks-engine/releases/tag/v0.73.0
- https://github.com/Azure/aks-engine-azurestack/releases/tag/v0.75.3
- https://github.com/Azure/aks-engine-azurestack/releases/tag/v0.76.0
- https://github.com/Azure/aks-engine-azurestack/releases/tag/v0.77.0
- https://github.com/Azure/aks-engine-azurestack/releases/tag/v0.78.0
- https://github.com/Azure/aks-engine-azurestack/releases/tag/v0.79.0
- https://github.com/Azure/aks-engine-azurestack/releases/tag/v0.80.2
Applies to version v0.67.0 of AKS engine.
This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Update planning
The AKS engine upgrade command fully automates the upgrade process of your cluster. It handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
Upgrade considerations
- Are you using the correct marketplace items, AKS Base Ubuntu 16.04-LTS or 18.04 Image Distro or AKS Base Windows Server for your version of AKS engine? You can find the versions in the section "Download new images and AKS engine".
- Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
- Are you using a reliable machine to run AKS engine and from which you're performing upgrade operations?
- If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
- If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
- Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
- No system updates or scheduled tasks are planned.
- Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Use the upgrade command
You must use the aks-engine upgrade command as described in Upgrade a Kubernetes cluster on Azure Stack Hub.
Upgrade interruptions
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster can take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu image and AKS engine.
As explained in the documentation for AKS engine for Azure Stack Hub, deploying a Kubernetes cluster requires:
- The aks-engine binary (required).
- AKS Base Ubuntu 16.04-LTS Image Distro (deprecated - no longer use, change in API model to use 18.04 instead).
- AKS Base Ubuntu 18.04-LTS Image Distro (required for Linux agents).
- AKS Base Windows Server Image Distro (required for Windows agents).
New versions of these images are available with this update:
Check the AKS engine and Azure Stack version mapping table for the required AKS base images.
Follow the instructions in Add the Azure Kubernetes Services (AKS) engine prerequisites to the Azure Stack Hub Marketplace.
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
Upgrading Kubernetes clusters created with the Ubuntu 16.04 distro
Starting with AKS Engine v0.67.0, the Ubuntu 16.04 distro is no longer a supported option, as the OS has reached its end-of-life. In order to upgrade a cluster, make sure to set the OS distro to aks-ubuntu-18.04 in your input API model, the one generated by aks-engine deploy and passed as input to aks-engine upgrade:
"masterProfile": {
"distro": "aks-ubuntu-18.04"
},
"agentPoolProfiles": [{
"distro": "aks-ubuntu-18.04"
}]
AKS engine and Azure Stack version mapping
| Azure Stack Hub version | AKS engine version |
|---|---|
| 1910 | 0.43.0, 0.43.1 |
| 2002 | 0.48.0, 0.51.0 |
| 2005 | 0.48.0, 0.51.0, 0.55.0, 0.55.4 |
| 2008 | 0.55.4, 0.60.1 |
| 2102 | 0.60.1, 0.63.0, 0.67.0 |
| 2108 | 0.63.0, 0.67.0 |
Kubernetes version upgrade path in AKS engine v0.67.0
You can find the current version and upgrade version for Azure Stack Hub in the following table. This table applies to the AKS engine cluster in Azure Stack Hub. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
| Current version | Upgrade available |
|---|---|
| 1.15.12 | 1.16.14, 1.16.15 |
| 1.16.14 | 1.16.15, 1.17.17 |
| 1.17.11, 1.17.17 | 1.18.18 |
| 1.18.15, 1.18.18 | 1.19.10 |
| 1.19.10 | 1.19.15, 1.20.11 |
| 1.20.6 | 1.20.11 |
In the API model JSON file, specify the release and version values under the orchestratorProfile section. For example, if you're planning to deploy Kubernetes 1.17.17, the following two values must be set (see the example kubernetes-azurestack.json):
- "orchestratorRelease": "1.17",
- "orchestratorVersion": "1.17.17"
AKS engine and corresponding image mapping
| AKS engine | AKS base image | Kubernetes versions | API model samples |
|---|---|---|---|
| v0.43.1 | AKS Base Ubuntu 16.04-LTS Image Distro, October 2019 (2019.10.24) | 1.15.5, 1.15.4, 1.14.8, 1.14.7 | |
| v0.48.0 | AKS Base Ubuntu 16.04-LTS Image Distro, March 2020 (2020.03.19) | 1.15.10, 1.14.7 | |
| v0.51.0 | AKS Base Ubuntu 16.04-LTS Image Distro, May 2020 (2020.05.13), AKS Base Windows Image (17763.1217.200513) | 1.15.12, 1.16.8, 1.16.9 | Linux, Windows |
| v0.55.0 | AKS Base Ubuntu 16.04-LTS Image Distro, August 2020 (2020.08.24), AKS Base Windows Image (17763.1397.200820) | 1.15.12, 1.16.14, 1.17.11 | Linux, Windows |
| v0.55.4 | AKS Base Ubuntu 16.04-LTS Image Distro, September 2020 (2020.09.14), AKS Base Windows Image (17763.1397.200820) | 1.15.12, 1.16.14, 1.17.11 | Linux, Windows |
| V0.60.1 | AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28), AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28), AKS Base Windows Image (17763.1697.210129) |
1.16.14, 1.16.15, 1.17.17, 1.18.15 | Linux, Windows |
| v0.63.0 | AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q2 (2021.05.24), AKS Base Windows Image (17763.1935.210520) | 1.18.18, 1.19.10, 1.20.6 | API Model Samples (Linux, Windows) |
| v0.67.0 | AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q3 (2021.09.27), AKS Base Windows Image (17763.2213.210927) | 1.19.15, 1.20.11 | API Model Samples (Linux, Windows) |
What's new
If you're interested in participating in a private preview, you can request preview access.
New features include:
- Support for Kubernetes 1.19.15 and 1.20.11
Known issues
- Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
- Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond those previously listed. You can ignore them and use the version table above as the official guide for this update.
Reference
The following list is the complete set of release notes for Azure and Azure Stack Hub combined:
- https://github.com/Azure/aks-engine/releases/tag/v0.64.0
- https://github.com/Azure/aks-engine/releases/tag/v0.65.0
- https://github.com/Azure/aks-engine/releases/tag/v0.65.1
- https://github.com/Azure/aks-engine/releases/tag/v0.66.0
- https://github.com/Azure/aks-engine/releases/tag/v0.66.1
- https://github.com/Azure/aks-engine/releases/tag/v0.67.0
Applies to version v0.60.1 of AKS engine.
This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Update planning
The AKS engine upgrade command fully automates the upgrade process of your cluster, and handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
Upgrade considerations
- Are you using the correct marketplace items, AKS Base Ubuntu 16.04-LTS or 18.04 Image Distro or AKS Base Windows Server for your version of AKS engine? You can find the versions in the section "Download new images and AKS engine".
- Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
- Are you using a reliable machine to run AKS engine and from which you're performing upgrade operations?
- If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
- If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
- Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
- No system updates or scheduled tasks are planned.
- Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Use the upgrade command
You must use the aks-engine upgrade command as described in Upgrade a Kubernetes cluster on Azure Stack Hub.
Upgrade interruptions
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu image and AKS engine.
As explained in the documentation for AKS engine on Azure Stack Hub, deploying a Kubernetes cluster requires:
- The aks-engine binary (required).
- AKS Base Ubuntu 16.04-LTS Image Distro (deprecated - no longer use, change in API model to use 18.04 instead).
- AKS Base Ubuntu 18.04-LTS Image Distro (required for Linux agents).
- AKS Base Windows Server Image Distro (required for Windows agents).
New versions of these are available with this update:
The Azure Stack Hub operator must download the new AKS Base images into the stamp marketplace:
AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28)
AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28),
AKS Base Windows Image (17763.1697.210129)
Follow the instructions in Add the Azure Kubernetes Services (AKS) engine prerequisites to the Azure Stack Hub Marketplace.
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
AKS engine and Azure Stack version mapping
| Azure Stack Hub version | AKS engine version |
|---|---|
| 1910 | 0.43.0, 0.43.1 |
| 2002 | 0.48.0, 0.51.0 |
| 2005 | 0.48.0, 0.51.0, 0.55.0, 0.55.4 |
| 2008 | 0.55.4, 0.60.1 |
Kubernetes version upgrade path in AKS engine v0.60.1
You can find the current version and upgrade version for Azure Stack Hub in the following table. This table applies to the AKS engine cluster in Azure Stack Hub. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
| Current version | Upgrade available |
|---|---|
| 1.15.12 | 1.16.14, 1.16.15 |
| 1.16.14 | 1.16.15, 1.17.17 |
| 1.17.11 | 1.17.17, 1.18.15 |
| 1.17.17 | 1.18.15 |
In the API model JSON file, specify the release and version values under the orchestratorProfile section. For example, if you're planning to deploy Kubernetes 1.17.17, the following two values must be set (see the example kubernetes-azurestack.json):
- "orchestratorRelease": "1.17",
- "orchestratorVersion": "1.17.17"
AKS engine and corresponding image mapping
| AKS engine | AKS base image | Kubernetes versions | API model samples |
|---|---|---|---|
| v0.43.1 | AKS Base Ubuntu 16.04-LTS Image Distro, October 2019 (2019.10.24) | 1.15.5, 1.15.4, 1.14.8, 1.14.7 | |
| v0.48.0 | AKS Base Ubuntu 16.04-LTS Image Distro, March 2020 (2020.03.19) | 1.15.10, 1.14.7 | |
| v0.51.0 | AKS Base Ubuntu 16.04-LTS Image Distro, May 2020 (2020.05.13), AKS Base Windows Image (17763.1217.200513) | 1.15.12, 1.16.8, 1.16.9 | Linux, Windows |
| v0.55.0 | AKS Base Ubuntu 16.04-LTS Image Distro, August 2020 (2020.08.24), AKS Base Windows Image (17763.1397.200820) | 1.15.12, 1.16.14, 1.17.11 | Linux, Windows |
| v0.55.4 | AKS Base Ubuntu 16.04-LTS Image Distro, September 2020 (2020.09.14), AKS Base Windows Image (17763.1397.200820) | 1.15.12, 1.16.14, 1.17.11 | Linux, Windows |
| V0.60.1 | AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28), AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28), AKS Base Windows Image (17763.1697.210129) |
1.16.14, 1.16.15, 1.17.17, 1.18.15 | Linux, Windows |
What's new
If you're interested in participating in a private preview, you can request preview access.
New features include:
- General Availability of Ubuntu 18.04.
- Certificate Rotation Public Preview #4214.
- T4 Nvidia GPU Private Preview #4259.
- Azure Active Directory integration private preview.
- CSI Driver for Azure Blobs Private Preview #712.
- CSI Driver Azure Disks Public Preview #712.
- CSI Driver NFS Public Preview #712.
- Support for Kubernetes 1. 17.17 #4188 and 1.18.15 #4187.
Known issues
- Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
- Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond those previously listed. You can ignore them and use the version table above as the official guide for this update.
Reference
The following list is the complete set of release notes for Azure and Azure Stack Hub combined:
Applies to version v0.55.4 of AKS engine.
This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Update planning
The AKS engine upgrade command fully automates the upgrade process of your cluster, and handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
Upgrade considerations
- Are you using the correct marketplace item, AKS Base Ubuntu 16.04-LTS Image Distro for your version of AKS engine? You can find the versions in the "Download new image and AKS engine" section.
- Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
- Are you using a reliable machine to run the AKS engine and from which you're performing upgrade operations?
- If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
- If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
- Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
- No system updates or scheduled tasks are planned.
- Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Use the upgrade command
You must use the aks-engine upgrade command as described in Upgrade a Kubernetes cluster on Azure Stack Hub.
Upgrade interruptions
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu Image and AKS engine.
As explained in the documentation for AKS engine on Azure Stack Hub, deploying a Kubernetes cluster requires two main components:
- The aks-engine binary (required).
- AKS Base Ubuntu 16.04-LTS Image Distro
New versions of these images are available with this update:
The Azure Stack Hub operator must download a new AKS base Ubuntu image into the stamp marketplace:
Name: AKS Base Ubuntu 16.04-LTS Image Distro, September 2020 (2020.09.14)
Version: 2020.09.14
Follow the instructions in Add the Azure Kubernetes Services (AKS) engine prerequisites to the Azure Stack Hub Marketplace.
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
AKS engine and Azure Stack version mapping
| Azure Stack Hub version | AKS engine version |
|---|---|
| 1910 | 0.43.0, 0.43.1 |
| 2002 | 0.48.0, 0.51.0 |
| 2005 | 0.48.0, 0.51.0, 0.55.0, 0.55.4 |
Kubernetes version upgrade path in AKS engine v0.55.4
You can find the current version and upgrade version for Azure Stack Hub in the following table. This table applies to the AKS engine cluster in Azure Stack Hub. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
| Current version | Upgrade available |
|---|---|
| 1.15.10 | 1.15.12 |
| 1.15.12, 1.16.8, 1.16.9 | 1.16.14 |
| 1.16.8, 1.16.9, 1.16.14 | 1.17.11 |
In the API model JSON file, specify the release and version values under the orchestratorProfile section. For example, if you're planning to deploy Kubernetes 1.17.17, the following two values must be set (see the example kubernetes-azurestack.json):
- "orchestratorRelease": "1.16",
- "orchestratorVersion": "1.16.14"
AKS engine and corresponding image mapping
| Kubernetes versions | Notes |
|---|---|
| 1.15.5, 1.15.4, 1.14.8, 1.14.7 | |
| 1.15.10, 1.14.7 | |
| 1.15.12, 1.16.8, 1.16.9 | API model samples (Linux, Windows) |
| 1.15.12, 1.16.14, 1.17.11 | API model samples (Linux, Windows) |
| 1.15.12, 1.16.14, 1.17.11 | API model samples (Linux, Windows) |
What's new
- Update Azure Stack's Linux VHD to 2020.09.14 #3828
- Adds support for K8s v1.17.11 on Azure Stack #3702
- Adds support for K8s v1.16.14 on Azure Stack #3704
- Linux VHD update to 2020.09.14 #3750
- Windows VHD update to August #3730
- Upgrades Kubernetes metrics-server to v0.3.7 #3669
- Upgrades docker version to fix log rotation issue #3693
- Upgrades CoreDNS to v1.7.0 #3608
- Use moby 19.03.x packages #3549
- Fixes to azure-cni update strategy #3571
Known issues
- Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
- When you run aks-engine
get-versions, the output produces information applicable to Azure and Azure Stack Hub; however, there's no explicit way to discern what corresponds to Azure Stack Hub. Don't use this command to determine what versions are available to upgrade. Use the upgrade reference table described in the previous section. - Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond those previously listed. You can ignore them and use the version table above as the official guide for this update.
Reference
This is the complete set of release notes for Azure and Azure Stack Hub combined:
- https://github.com/Azure/aks-engine/releases/tag/v0.51.1
- https://github.com/Azure/aks-engine/releases/tag/v0.52.1
- https://github.com/Azure/aks-engine/releases/tag/v0.53.1
- https://github.com/Azure/aks-engine/releases/tag/v0.54.1
- https://github.com/Azure/aks-engine/releases/tag/v0.55.0
- https://github.com/Azure/aks-engine/releases/tag/v0.55.4
Applies to version 0.48.0 or earlier of AKS engine.
This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. This article isn't intended to document the release information for AKS engine for global Azure.
Update planning
The AKS engine upgrade command fully automates the upgrade process of your cluster, and handles virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.
Upgrade considerations
- Are you using the correct marketplace item, AKS Base Ubuntu 16.04-LTS Image Distro for your version of AKS engine? You can find the versions in the section Download the new AKS base Ubuntu Image and AKS engine versions.
- Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
- Are you using a reliable machine to run AKS engine and from which you're performing upgrade operations?
- If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
- If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
- Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a few more VMs to be created during the process.
- No system updates or scheduled tasks are planned.
- Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.
Use the upgrade command
You must use the aks-engine upgrade command as described in Upgrade a Kubernetes cluster on Azure Stack Hub.
Upgrade interruptions
Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and re-submit the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once you resubmit the command. Normally, interruptions increase the time to complete the update, but shouldn't affect its completion.
Estimated upgrade time
The estimated upgrade time is 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take about 5 hours to upgrade.
Download new image and AKS engine
Download the new versions of the AKS base Ubuntu Image and AKS engine.
As explained in the documentation for AKS engine on Azure Stack Hub, deploying a Kubernetes cluster requires two main components:
- The aks-engine binary (required).
- AKS Base Ubuntu 16.04-LTS Image Distro.
New versions of these are available with this update:
The Azure Stack Hub operator must download the new AKS Base images into the stamp marketplace:
Name:
AKS Base Ubuntu 16.04-LTS Image Distro, March 2020Version:
2020.03.19Follow the instructions in Add the Azure Kubernetes Services (AKS) engine prerequisites to the Azure Stack Hub Marketplace.
The Kubernetes cluster administrator (normally a tenant user of Azure Stack Hub) must download the new aks-engine. See the instructions in Install the AKS engine on Linux in Azure Stack Hub (or equivalent Windows article). You can follow the same process you used to install the cluster for the first time. The update overwrites the previous binary. For example, if you used the get-akse.sh script, follow the same steps outlined in Install in a connected environment. The same process applies if you're installing on a Windows system: Install the AKS engine on Windows in Azure Stack Hub.
Kubernetes version upgrade path
You can find the current version and upgrade version for Azure Stack Hub in the following table. This table applies to the AKS engine cluster in Azure Stack Hub. Don't use the aks-engine get-versions command, which returns versions supported in global Azure as well as in Azure Stack Hub.
| Current version | Upgrade available |
|---|---|
| 1.14.7 | 1.15.10 |
| 1.14.8 | 1.15.10 |
| 1.15.4 | 1.15.10 |
| 1.15.5 | 1.15.10 |
What's new
Support for Kubernetes version 1.15.10 (#2834). When you deploy a new cluster, in your API model .json file (also known as the cluster definition file), specify both the release version number and the minor version number in the following format. For an example cluster definition file, see kubernetes-azurestack.json.
"orchestratorRelease": "1.15,"orchestratorVersion": "1.15.10"
Note
If the Kubernetes version is not explicitly provided in the API model .json file, version 1.15 will be used (#2932) and the orchestratorVersion will default to 1.15.11, which will result in an error during deployment of the cluster.
With aks-engine v0.43.1, the default frequency settings for the cloud provider to perform its control loop and other tasks don't work well with Azure Stack Hub Resource Manager threshold limits for incoming requests. This update changes defaults for Azure Stack Hub to reduce the retry load to Azure Stack Hub Resource Manager (#2861).
New verification step in aks-engine will result in either execution stopping or displaying warnings if API model .json file contains properties not supported by Azure Stack Hub (#2717).
With a new verification check-in, the aks-engine will validate availability of the version of the AKS base image needed for the version of aks-engine executing (#2342). This will occur after parsing the API model .json file and before calling the Azure Stack Hub Resource Manager.
New aks-engine option "--control-plane-only" in the
upgradecommand allows the user to upgrade operations to target only the master Virtual Machines (#2635).Updates to Linux Kernel version 4.15.0-1071-azure for Ubuntu 16.04-LTS. See "Package: linux-image-4.15.0-1071-azure (4.15.0-1071.76) [security]" for details.
New hyperkube updates to support Kubernetes versions 1.14.8 and 1.15.10.
Update kubectl to match the version of Kubernetes for the cluster. This component is available in the Kubernetes cluster control plane nodes, you can run it by SSH into a master.
Updates for the Azure Container Monitor add-in with latest February 2020 release (#2850).
Upgrade of
corednsto version v1.6.6 (#2555).Upgrade
etcdto version 3.3.18 (#2462).Upgrade
mobyto version 3.0.11 (#2887).With this release AKS Engine cuts dependency from
k8s.gcr.ioto now use the officialKubernetes MCR registry @ mcr.microsoft.comwhen building its images (#2722).
Known issues
- Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at a time.
- Running aks-engine
get-versionsproduces information applicable to Azure and Azure Stack Hub; however, there's no explicit way to discern what corresponds to Azure Stack Hub. Don't use this command to determine what versions are available to upgrade. Use the upgrade reference table described in the previous section. - Since the aks-engine tool is a shared source code repository across Azure and Azure Stack Hub, examining the many release notes and pull requests might lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond those previously listed. You can ignore them and use the version table above as the official guide for this update.
Reference
The following is a list of some of the bugs fixed and the complete set of release notes from version 0.44.0 to version 0.48.0. The release notes include both Azure and Azure Stack Hub.
Bug fixes
userAssignedIdentityIdin Windows azure.json missing quotes (#2327)Add-ins
update configis upgrade-only (#2282)Bumping timeout for getting management IP on Windows nodes (#2284)
Add 1.0.28 Azure CNI .zip file into Windows VHD (#2268)
Correct defaults order for setting
IPAddressCount(#2358)Update to use single omsagent yaml for all K8s versions to avoid any manual errors and easy maintainability (#2692)
Release notes
This is the complete set of release notes for Azure and Azure Stack Hub combined:
- https://github.com/Azure/aks-engine/releases/tag/v0.44.0
- https://github.com/Azure/aks-engine/releases/tag/v0.44.1
- https://github.com/Azure/aks-engine/releases/tag/v0.44.2
- https://github.com/Azure/aks-engine/releases/tag/v0.45.0
- https://github.com/Azure/aks-engine/releases/tag/v0.46.0
- https://github.com/Azure/aks-engine/releases/tag/v0.46.1
- https://github.com/Azure/aks-engine/releases/tag/v0.46.2
- https://github.com/Azure/aks-engine/releases/tag/v0.46.3
- https://github.com/Azure/aks-engine/releases/tag/v0.47.0
- https://github.com/Azure/aks-engine/releases/tag/v0.48.0
Next steps
- Read about AKS engine on Azure Stack Hub