how use azure vault with managed identity on azure arc enabled k8s cluster.

Question

we have follow the given step to install and use azure vault with our on-prem cluster.

so we want to use azure managed identity with our on-prem cluster thats why we use azure arc

to connect our cluster to azure.

steps:

1. connect on-prem cluster to azure:

• az extenstion add --name connectedk8s
• az provider register -- namespace Microsoft.Kubernetes
• az provider register -- namespace Microsoft.KubernetesConfiguration
• az provider register -- namespace Microsoft.ExtendedLocation
• az connectedk8s connect --name <cluster-name> --resource-group <resource-group-name> --location <location>

2.create kubernetes serviceaccount and clusterrolebinding:

• kubectl create serviceaccount azure-arc-viewer -n azure-arc
• kubectl create clusterrolebinding azure-arc-viewer-binding --clusterrole=view --serviceaccount=azure-arc:azure-arc-viewer

3.genrate kubernetes token for azure arc:

• kubectl create token azure-arc-viewer
• kubectl get secret azure-arc-viewer-token -n azure-arc -o jsonpath="{.data.token}" | base64 --decode

cluster successfully connected to azure arc.

we have aleady create a sample secret in vault. now we want to use it in our on-prem cluster with managed identity.

4.Install Secrets Store CSI Driver & Azure Key Vault Provider:

az k8s-extension create \

--name akvsecretsprovider \

--cluster-name <your-arc-cluster-name> \

--resource-group <your-resource-group> \

--cluster-type connectedClusters \

--extension-type Microsoft.AzureKeyVaultSecretsProvider \

--config auto-rotate-secrets=true

5.verify the installation:

kubectl get po -n kube-system

pods are running successfully.

6.Grant Cluster Access to Azure Key Vault

• Get your cluster's Managed Identity ID:

az connectedk8s show --name <cluster-name> --resource-group <resource-group-name> --query identity.principalId -o tsv

create azure identity and assign role to it:

• Navigate to Your Key Vault
• Click on Access Control (IAM)
• Click on Add Role Assignment
• Select the Role as Key Vault Secrets User
• Assign access to: Choose Managed identity.
• Managed identity: Select Azure Arc enabled Kubernetes cluster
• Review + assign: Confirm and click Review + assign.

7.for testing we create secretproviderclass and test pod:

apiVersion: secrets-store.csi.x-k8s.io/v1

kind: SecretProviderClass

metadata:

  name: azure-kv-secrets

spec:

  provider: azure

  parameters:

    usePodIdentity: "false"

    useVMManagedIdentity: "true"           

    userAssignedIdentityID: ""              

    keyvaultName: <your-key-vault-name>

    objects: |

      array:

        - |

          objectName: my-secret            

          objectType: secret              

    tenantId: <your-azure-tenant-id>       

test pod:

apiVersion: v1

kind: Pod

metadata:

  name: nginx-secrets-test

spec:

  containers:

  - name: nginx

    image: nginx

    volumeMounts:

    - name: secrets-store

      mountPath: "/mnt/secrets"

      readOnly: true

  volumes:

    - name: secrets-store

      csi:

        driver: secrets-store.csi.k8s.io

        readOnly: true

        volumeAttributes:

          secretProviderClass: "azure-kv-secrets"

still its show the error.

`
if seVMManagedIdentity = true in SecretProviderClass

Normal Scheduled 2m11s default-scheduler Successfully assigned default/nginx-secrets-test to 10.171.248.85

Warning FailedMount 11s kubelet MountVolume.SetUp failed for volume "secrets-store" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

if seVMManagedIdentity = false in SecretProviderClass
Normal Scheduled 16s default-scheduler Successfully assigned default/nginx-secrets-test to 10.171.249.13

Warning FailedMount 1s (x6 over 17s) kubelet MountVolume.SetUp failed for volume "secrets-store" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/nginx-secrets-test, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set

`

can someone help