Prepare your Azure Arc-enabled Kubernetes cluster

Article
08/06/2024

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in preview. You shouldn't use this preview software in production environments.

You'll need to deploy a new Azure IoT Operations installation when a generally available release is made available. You won't be able to upgrade a preview installation.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

An Azure Arc-enabled Kubernetes cluster is a prerequisite for deploying Azure IoT Operations Preview. This article describes how to prepare an Azure Arc-enabled Kubernetes cluster before you Deploy Azure IoT Operations Preview to an Arc-enabled Kubernetes cluster to run your own workloads. This article includes guidance for both Ubuntu, Windows, and cloud environments.

Tip

If you want to deploy Azure IoT Operations and run a sample workload, see the Quickstart: Run Azure IoT Operations Preview in Github Codespaces with K3s.

Azure IoT Operations ships as a set of Azure Arc-enabled Kubernetes services and is intended for use with CNCF conformant Arc validated partner products. Currently, Microsoft has validated Azure IoT Operations against the following fixed-set of infrastructures and environments:

Environment	Version
AKS-EE on Windows 11 IoT Enterprise on a single-node AMD Ryzen-7 (8 core, 3.3 GHz), 16-GB RAM	AksEdge-K3s-1.28.3-1.7.639.0
K3s on Ubuntu 22.04.2 on a single-node AMD Ryzen-7 (8 core, 3.3 GHz), 16-GB RAM	K3s version 1.28.5

Important

The environments listed previously are production-like environments that Microsoft has validated. They're not the only environments that Azure IoT Operations can run on. Azure IoT Operations can run on any Arc-enabled Kubernetes cluster that meets the Azure Arc-enabled Kubernetes system requirements.

Prerequisites

To prepare your Azure Arc-enabled Kubernetes cluster, you need:

Hardware that meets the system requirements.

An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.
Azure CLI version 2.46.0 or newer installed on your development machine. Use az --version to check your version and az upgrade to update if necessary. For more information, see How to install the Azure CLI.
The Azure IoT Operations extension for Azure CLI. Use the following command to add the extension or update it to the latest version:
```
az extension add --upgrade --name azure-iot-ops
```
Hardware that meets the system requirements:
- Ensure that your machine has a minimum of 10-GB RAM, 4 vCPUs, and 40-GB free disk space.
- Review the AKS Edge Essentials requirements and support matrix.
- Review the AKS Edge Essentials networking guidance.

An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.
Azure CLI version 2.46.0 or newer installed on your development machine. Use az --version to check your version and az upgrade to update if necessary. For more information, see How to install the Azure CLI.
The Azure IoT Operations extension for Azure CLI. Use the following command to add the extension or update it to the latest version:
```
az extension add --upgrade --name azure-iot-ops
```
Review the K3s requirements.

Azure IoT Operations also works on Ubuntu in Windows Subsystem for Linux (WSL) on your Windows machine. Use WSL for testing and development purposes only.

To set up your WSL Ubuntu environment:

Install Linux on Windows with WSL.
Enable systemd:
```
sudo -e /etc/wsl.conf
```
Add the following to wsl.conf and then save the file:
```
[boot]
systemd=true
```

After you enable systemd, re-enable running windows executables from WSL:

sudo sh -c 'echo :WSLInterop:M::MZ::/init:PF > /usr/lib/binfmt.d/WSLInterop.conf'
sudo systemctl unmask systemd-binfmt.service
sudo systemctl restart systemd-binfmt
sudo systemctl mask systemd-binfmt.service

Create a cluster

This section provides steps to prepare and Arc-enable clusters in validated environments on Linux and Windows as well as GitHub Codespaces in the cloud.

Azure Kubernetes Service Edge Essentials is an on-premises Kubernetes implementation of Azure Kubernetes Service (AKS) that automates running containerized applications at scale. AKS Edge Essentials includes a Microsoft-supported Kubernetes platform that includes a lightweight Kubernetes distribution with a small footprint and simple installation experience, making it easy for you to deploy Kubernetes on PC-class or "light" edge hardware.

The AksEdgeQuickStartForAio.ps1 script automates the the process of creating and connecting a cluster, and is the recommended path for deploying Azure IoT Operations on AKS Edge Essentials.

Open an elevated PowerShell window and change the directory to a working folder.

Run the following commands, replacing the placeholder values with your information:

Placeholder	Value
SUBSCRIPTION_ID	The ID of your Azure subscription. If you don't know your subscription ID, see Find your Azure subscription.
TENANT_ID	The ID of your Microsoft Entra tenant. If you don't know your tenant ID, see Find your Microsoft Entra tenant.
RESOURCE_GROUP_NAME	The name of an existing resource group or a name for a new resource group to be created.
LOCATION	An Azure region close to you. For the list of currently supported Azure regions, see Supported regions.
CLUSTER_NAME	A name for the new cluster to be created.

$url = "https://raw.githubusercontent.com/Azure/AKS-Edge/main/tools/scripts/AksEdgeQuickStart/AksEdgeQuickStartForAio.ps1"
Invoke-WebRequest -Uri $url -OutFile .\AksEdgeQuickStartForAio.ps1
Unblock-File .\AksEdgeQuickStartForAio.ps1
Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process -Force
.\AksEdgeQuickStartForAio.ps1 -SubscriptionId "<SUBSCRIPTION_ID>" -TenantId "<TENANT_ID>" -ResourceGroupName "<RESOURCE_GROUP_NAME>"  -Location "<LOCATION>"  -ClusterName "<CLUSTER_NAME>"

If there are any issues during deployment, including if your machine reboots as part of this process, run the whole set of commands again.

Run the following commands to check that the deployment was successful:
```
Import-Module AksEdge
Get-AksEdgeDeploymentInfo
```
In the output of the Get-AksEdgeDeploymentInfo command, you should see that the cluster's Arc status is Connected.

Azure IoT Operations should work on any CNCF-conformant kubernetes cluster. For Ubuntu Linux, Microsoft currently supports K3s clusters.

Important

If you're using Ubuntu in Windows Subsystem for Linux (WSL), run all of these steps in your WSL environment, including the Azure CLI steps for configuring your cluster.

To prepare a K3s Kubernetes cluster on Ubuntu:

Run the K3s installation script:
```
curl -sfL https://get.k3s.io | sh -
```
For full installation information, see the K3s quick-start guide.

Create a K3s configuration yaml file in .kube/config:

mkdir ~/.kube
sudo KUBECONFIG=~/.kube/config:/etc/rancher/k3s/k3s.yaml kubectl config view --flatten > ~/.kube/merged
mv ~/.kube/merged ~/.kube/config
chmod  0600 ~/.kube/config
export KUBECONFIG=~/.kube/config
#switch to k3s context
kubectl config use-context default

Run the following command to increase the user watch/instance limits.

echo fs.inotify.max_user_instances=8192 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf

sudo sysctl -p

For better performance, increase the file descriptor limit:

echo fs.file-max = 100000 | sudo tee -a /etc/sysctl.conf

sudo sysctl -p

Important

Codespaces are easy to set up quickly and tear down later, but they're not suitable for performance evaluation or scale testing. Use GitHub Codespaces for exploration only.

Use GitHub Codespaces to try Azure IoT Operations on a Kubernetes cluster without installing anything on your local machine. The Azure-Samples/explore-iot-operations codespace is preconfigured with:

K3s running in K3d for a lightweight Kubernetes cluster
Azure CLI
Kubectl for managing Kubernetes resources
Other useful tools like Helm and k9s

To create your codespace and cluster, use the following steps:

Create a codespace in GitHub Codespaces.

Provide the following recommended secrets for your codespace:

Parameter	Value
SUBSCRIPTION_ID	Your Azure subscription ID.
RESOURCE_GROUP	A name for a new Azure resource group where your cluster will be created.
LOCATION	An Azure region close to you. For the list of currently supported regions, see Supported regions.

Tip

The values you provide as secrets in this step get saved on your GitHub account to be used in this and future codespaces. They're also automatically added as environment variables in the codespace terminal, and you can use those environment variables in the CLI commands in the next section.

Additionally, this codespace automatically creates a CLUSTER_NAME environment variable which is set with the codespace name.

Select Create new codespace.
Once the codespace is ready, select the menu button at the top left, then select Open in VS Code Desktop.
If prompted, install the GitHub Codespaces extension for Visual Studio Code and sign in to GitHub.
In Visual Studio Code, select View > Terminal.

Use this terminal to run all of the command line and CLI commands for managing your cluster.

Arc-enable your cluster

Connect your cluster to Azure Arc so that it can be managed remotely.

The AksEdgeQuickStartForAio.ps1 script that you ran in the previous section handled the steps to connect your cluster. You don't need to take any extra steps to Arc-enable.

To connect your cluster to Azure Arc:

On the machine where you deployed the Kubernetes cluster, or in your WSL environment, sign in with Azure CLI:
```
az login
```

Set environment variables for your Azure subscription, location, a new resource group, and the cluster name as it will show up in your resource group.

For the list of currently supported Azure regions, see Supported regions.

# Id of the subscription where your resource group and Arc-enabled cluster will be created
export SUBSCRIPTION_ID=<SUBSCRIPTION_ID>

# Azure region where the created resource group will be located
export LOCATION=<REGION>

# Name of a new resource group to create which will hold the Arc-enabled cluster and Azure IoT Operations resources
export RESOURCE_GROUP=<NEW_RESOURCE_GROUP_NAME>

# Name of the Arc-enabled cluster to create in your resource group
export CLUSTER_NAME=<NEW_CLUSTER_NAME>

Set the Azure subscription context for all commands:
```
az account set -s $SUBSCRIPTION_ID
```
Register the required resource providers in your subscription:

Note

This step only needs to be run once per subscription. To register resource providers, you need permission to do the /register/action operation, which is included in subscription Contributor and Owner roles. For more information, see Azure resource providers and types.
```
az provider register -n "Microsoft.ExtendedLocation"
az provider register -n "Microsoft.Kubernetes"
az provider register -n "Microsoft.KubernetesConfiguration"
az provider register -n "Microsoft.IoTOperationsOrchestrator"
az provider register -n "Microsoft.IoTOperations"
az provider register -n "Microsoft.DeviceRegistry"
```
Use the az group create command to create a resource group in your Azure subscription to store all the resources:
```
az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```
Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it as part of your Azure resource group:
```
az connectedk8s connect -n $CLUSTER_NAME -l $LOCATION -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```
Get the objectId of the Microsoft Entra ID application that the Azure Arc service uses and save it as an environment variable.
```
export OBJECT_ID=$(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv)
```
Use the az connectedk8s enable-features command to enable custom location support on your cluster. This command uses the objectId of the Microsoft Entra ID application that the Azure Arc service uses. Run this command on the machine where you deployed the Kubernetes cluster:
```
az connectedk8s enable-features -n $CLUSTER_NAME -g $RESOURCE_GROUP --custom-locations-oid $OBJECT_ID --features cluster-connect custom-locations
```

To connect your cluster to Azure Arc:

In your codespace terminal, sign in to Azure CLI:
```
az login
```
Tip

If you're using the GitHub codespace environment in a browser rather than VS Code desktop, running az login returns a localhost error. To fix the error, either:
- Open the codespace in VS Code desktop, and then return to the browser terminal and rerun az login.
- Or, after you get the localhost error on the browser, copy the URL from the browser and run curl "<URL>" in a new terminal tab. You should see a JSON response with the message "You have logged into Microsoft Azure!."
Set the Azure subscription context for all commands:
```
az account set -s $SUBSCRIPTION_ID
```
Register the required resource providers in your subscription:

Note

This step only needs to be run once per subscription. To register resource providers, you need permission to do the /register/action operation, which is included in subscription Contributor and Owner roles. For more information, see Azure resource providers and types.
```
az provider register -n "Microsoft.ExtendedLocation"
az provider register -n "Microsoft.Kubernetes"
az provider register -n "Microsoft.KubernetesConfiguration"
az provider register -n "Microsoft.IoTOperationsOrchestrator"
az provider register -n "Microsoft.IoTOperations"
az provider register -n "Microsoft.DeviceRegistry"
```
Use the az group create command to create a resource group in your Azure subscription to store all the resources:
```
az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```
Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it as part of your Azure resource group:
```
az connectedk8s connect -n $CLUSTER_NAME -l $LOCATION -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```
Tip

The value of $CLUSTER_NAME is automatically set to the name of your codespace. Replace the environment variable if you want to use a different name.
Get the objectId of the Microsoft Entra ID application that the Azure Arc service uses and save it as an environment variable.
```
export OBJECT_ID=$(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv)
```
Use the az connectedk8s enable-features command to enable custom location support on your cluster. This command uses the objectId of the Microsoft Entra ID application that the Azure Arc service uses.
```
az connectedk8s enable-features -n $CLUSTER_NAME -g $RESOURCE_GROUP --custom-locations-oid $OBJECT_ID --features cluster-connect custom-locations
```

Verify your cluster

To verify that your cluster is ready for Azure IoT Operations deployment, you can use the verify-host helper command in the Azure IoT Operations extension for Azure CLI. When run on the cluster host, this helper command checks connectivity to Azure Resource Manager and Microsoft Container Registry endpoints.

az iot ops verify-host

To verify that your Kubernetes cluster is now Azure Arc-enabled, run the following command:

kubectl get deployments,pods -n azure-arc

The output looks like the following example:

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/clusterconnect-agent         1/1     1            1           10m
deployment.apps/extension-manager            1/1     1            1           10m
deployment.apps/clusteridentityoperator      1/1     1            1           10m
deployment.apps/controller-manager           1/1     1            1           10m
deployment.apps/flux-logs-agent              1/1     1            1           10m
deployment.apps/cluster-metadata-operator    1/1     1            1           10m
deployment.apps/extension-events-collector   1/1     1            1           10m
deployment.apps/config-agent                 1/1     1            1           10m
deployment.apps/kube-aad-proxy               1/1     1            1           10m
deployment.apps/resource-sync-agent          1/1     1            1           10m
deployment.apps/metrics-agent                1/1     1            1           10m

NAME                                              READY   STATUS    RESTARTS        AGE
pod/clusterconnect-agent-5948cdfb4c-vzfst         3/3     Running   0               10m
pod/extension-manager-65b8f7f4cb-tp7pp            3/3     Running   0               10m
pod/clusteridentityoperator-6d64fdb886-p5m25      2/2     Running   0               10m
pod/controller-manager-567c9647db-qkprs           2/2     Running   0               10m
pod/flux-logs-agent-7bf6f4bf8c-mr5df              1/1     Running   0               10m
pod/cluster-metadata-operator-7cc4c554d4-nck9z    2/2     Running   0               10m
pod/extension-events-collector-58dfb78cb5-vxbzq   2/2     Running   0               10m
pod/config-agent-7579f558d9-5jnwq                 2/2     Running   0               10m
pod/kube-aad-proxy-56d9f754d8-9gthm               2/2     Running   0               10m
pod/resource-sync-agent-769bb66b79-z9n46          2/2     Running   0               10m
pod/metrics-agent-6588f97dc-455j8                 2/2     Running   0               10m

Create sites

A site is a collection of Azure IoT Operations instances. Sites typically group instances by physical location and make it easier for OT users to locate and manage assets. An IT administrator creates sites and assigns Azure IoT Operations instances to them. To learn more, see What is Azure Arc site manager (preview)?.

Next steps

Now that you have an Azure Arc-enabled Kubernetes cluster, you can deploy Azure IoT Operations.

Share via

Prepare your Azure Arc-enabled Kubernetes cluster

Prerequisites

Create a cluster

Arc-enable your cluster

Verify your cluster

Create sites

Next steps

Feedback

Additional resources