AKS cannot pull image from ACR with kubelet identity - 401 Unauthorized

Question

After using service principal for AKS for a year, I switched to user-assigned managed identity. Everything is working well except pulling image from ACR, here is the error:

Failed to pull image "myacr.azurecr.io/fluent/fluent-bit:3.1.6":[rpc error: code = Unknown desc = failed to pull and unpack image "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to resove reference "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://myacr.azurecr.io/oauth2/token?scope=repository%3Afluent%2Ffluent-bit%3Apull&service=myacr.azurecr.io: 401 Unauthorized, rpc error: code = Unknown desc = failed to pull and unpack image "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to resolve reference "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://myacr.azurecr.io/oauth2/token?scope=repository%3Afluent%2Ffluent-bit%3Apull&service=myacr.azurecr.io: 401 Unauthorized]

My resources meet all the requirements:

• AKS already have kubelet identity assigned.
• Kubelet identity was assigned AcrPull role with scope of exactly ACR.
• Kubelet identity was associated with VMSS of AKS node pools.
• There is no connection issue between ACR and AKS (It worked if I use secret in AKS)
• Service principal client id on AKS is msi.

I tried to follow these documents:

• https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/extensions/cannot-pull-image-from-acr-to-aks-cluster
• https://learn.microsoft.com/en-us/azure/aks/use-managed-identity

However, it didn't work. I even tried to scale new node on node pool.

It seems like this is a bug, anyone have solution for this?

Answer 1

Hi QuyHoang,

Thank you for reaching out to the Microsoft Q&A platform.

Encountering a 401 Unauthorized error when pulling images from Azure Container Registry (ACR) using a user-assigned managed identity in Azure Kubernetes Service (AKS).

Here are a few steps to troubleshoot and resolve this issue:

verify if the kubelet identity is correctly assigned, run the following command:

az aks show --resource-group <nameofResourcegrp> --name <cluster-name> --query "identityProfile.kubeletidentity"

After confirming the kubelet identity, check if the required role is assigned:

az role assignment list --assignee <your-managed-identity-client-id> --scope /subscriptions/<your-subscription-id>/resourceGroups/<your-resource-group>/providers/Microsoft.ContainerRegistry/registries/<your-acr-name>

Create Role Assignment (if missing) if the role is not assigned, you can create it with:

az role assignment create --assignee <your-managed-identity-client-id> --role AcrPull --scope /subscriptions/<your-subscription-id>/resourceGroups/<your-resource-group>/providers/Microsoft.ContainerRegistry/registries/<your-acr-name>

Update AKS Cluster Ensure the AKS cluster is updated to use the correct managed identity:

az aks update --resource-group <your-resource-group> --name <your-cluster-name> --assign-identity <your-managed-identity-resource-id>

Verify Kubelet Identity for Node Pools to check if the kubelet identity is applied to all node pools, use:

az vmss identity show --resource-group <your-resource-group> --name <your-node-pool-name>

Set Up Diagnostic Settings for ACR Optionally, you can create diagnostic settings for your Azure Container Registry (ACR):

az acr diagnostic-settings create --registry-name <your-acr-name> --logs Read --metrics Read --storage-account <your-storage-account>

If an answer has been helpful, please consider accept the answer and "Upvote" to help increase visibility of this question for other members of the Microsoft Q&A community. 

Thank you.