Edit

Share via

Resolve "az aks command invoke" failures

  • Article

This article describes how to resolve az aks command invoke failures in Microsoft Azure CLI so that you can successfully connect to any Azure Kubernetes Service (AKS) cluster, especially to a private AKS cluster.

Other connection methods need to use extra configuration components, as shown in the following table.

Connection methods Extra configuration component
Virtual network Virtual private network (VPN)
Peered network Azure ExpressRoute
Private endpoint Jumpbox

The az aks command invoke Azure CLI command is an alternative way of connecting to a cluster that doesn't require extra configuration components.

When you run the az aks command invoke command, Azure CLI automatically creates a command-<ID> pod in the aks-command namespace to access the AKS cluster and retrieve the required information.

Prerequisites

Symptoms

The following table lists common az aks command invoke error messages. Each error message has a link to the section that describes why the error is occurring, and how to fix it.

Error message Link
Operation returned an invalid status 'Not Found' Cause 1: The pod can't be created because of node or resource constraints
Failed to run command in managed cluster due to kubernetes failure. details: admission webhook "validation.gatekeeper.sh" denied the request: <policy-specific-message> Cause 2: Azure Policy doesn't allow the pod creation
Error from server (Forbidden): namespaces is forbidden: User "<ID>" cannot list resource "<resource>" in API group "" at the cluster scope Cause 3: Required roles aren't granted
Failed to connect to MSI. Please make sure MSI is configured correctly.

Get Token request returned: Response [400];		 Cause 4: There's a Cloud Shell issue

Cause 1: The pod can't be created because of node or resource constraints

The operation returns a Not Found status because the command-<ID> pod can't reach a successful state, such as Running. (In many cases, the pod stays in the Pending state.) In this case, the nodes aren't able to schedule the pod. This scenario can have different causes, such as the following causes:

  • Resource constraints
  • Nodes that have a NotReady or SchedulingDisabled state
  • Nodes that have taints that the pod can't tolerate
  • Other causes

Solution 1: Change the configuration so that you can schedule and run the pod

Make sure that the command-<ID> pod can be scheduled and run by adjusting the configuration. For example:

  • Increase the node pool size and make sure it has no pod secluding constraints like taints so that the command-<ID> pod can be deployed.
  • Adjust resource requests and limits in your pod specifications.

Cause 2: Azure Policy doesn't allow the pod creation

If you have specific Azure policies, the az aks command invoke command can fail because of a disallowed configuration in the command-<ID> pod. For example, you might have an Azure policy that requires a read-only root file system or other specific configuration.

Solution 2: Exempt the namespace for policies that prohibit pod creation

We recommend that you exempt the aks-command namespace for the associated Azure policies that don't allow the pod creation. For more information about exemption, see Understand scope in Azure Policy

To exempt an Azure Policy:

  1. In the Azure portal, search for and select Policy.

  2. In the Policy navigation pane, locate the Authoring section, and then select Assignments.

  3. In the table of assignments, find the row that contains the Assignment name that you want to change, and then select the name of the assignment.

  4. In the policy assignment page for that assignment, select Edit assignment.

  5. Select the Parameters tab.

  6. Clear the Only show parameters that need input or review option.

  7. In the Namespace exclusions box, add the aks-command namespace to the list of namespaces to be excluded.

Alternatively, if the policy isn't a built-in policy, you can check the configuration of the command-<ID> pod, and adjust the policy as necessary. To explore the pod's YAML configuration, run the following command:

kubectl get pods command-<ID> --namespace aks-command --output yaml

You can exempt the aks-command namespace from restrictive policies by running the following command:

az policy exemption create --name ExemptAksCommand --scope /subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.ContainerService/managedClusters/{aks-cluster} --policyAssignment /subscriptions/{subscription-id}/providers/Microsoft.Authorization/policyAssignments/{policy-assignment-id}

Cause 3: Required roles aren't granted

To use the az aks command invoke command, you must have access to the following roles on the cluster:

  • Microsoft.ContainerService/managedClusters/runCommand/action
  • Microsoft.ContainerService/managedClusters/commandResults/read

If you don't have these roles, the az aks command invoke command can't retrieve the required information.

Solution 3: Add the required roles

To resolve this issue, follow these steps:

  1. Add the Microsoft.ContainerService/managedClusters/runCommand/action and Microsoft.ContainerService/managedClusters/commandResults/read roles.

  2. Assign the necessary roles to the user:

    az role assignment create --assignee {user-principal-name} --role "Azure Kubernetes Service Cluster User Role" --scope /subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.ContainerService/managedClusters/{aks-cluster}

Cause 4: There's a Cloud Shell issue

The az aks command invoke command isn't processed as expected when it's run directly in the Azure Cloud Shell environment. This is a known issue in Cloud Shell.

Solution 4a: Run the az login command first

In Cloud Shell, run the az login command before you run the az aks command invoke command. For example:

az login
az aks command invoke --resource-group {resource-group} --name {aks-cluster} --command "kubectl get pods"

Solution 4b: Run the command on a local computer or a virtual machine

Run the az aks command invoke command on a local computer or any virtual machine (VM) that has Azure CLI installed.

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.