Use the Secret Store extension to fetch secrets for offline access in Azure Arc-enabled Kubernetes clusters

Article
12/10/2024

The Azure Key Vault Secret Store extension for Kubernetes ("SSE") automatically synchronizes secrets from an Azure Key Vault to an Azure Arc-enabled Kubernetes cluster for offline access. This means you can use Azure Key Vault to store, maintain, and rotate your secrets, even when running your Kubernetes cluster in a semi-disconnected state. Synchronized secrets are stored in the cluster secret store, making them available as Kubernetes secrets to be used in all the usual ways: mounted as data volumes, or exposed as environment variables to a container in a pod.

Synchronized secrets are critical business assets, so the SSE secures them through isolated namespaces and nodes, role-based access control (RBAC) policies, and limited permissions for the secrets synchronizer. For extra protection, encrypt the Kubernetes secret store on your cluster.

Tip

The SSE is recommended for scenarios where offline access is necessary, or if you need secrets synced into the Kubernetes secret store. If you don't need these features, you can use the Azure Key Vault Secrets Provider extension for secret management in your Arc-enabled Kubernetes clusters. It is not recommended to run both the online Azure Key Vault Secrets Provider extension and the offline SSE side-by-side in a cluster.

This article shows you how to install and configure the SSE as an Azure Arc-enabled Kubernetes extension.

Important

SSE is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Prerequisites

An Arc-enabled cluster. This can be one that you connected to yourself (the examples throughout this guide use a K3s cluster) or a Microsoft-managed AKS enanabled by Azure Arc cluster. The cluster must be running Kubernetes version 1.27 or higher, and in one of the supported regions (East US, East US2, West US, West US2, West US3, West Europe, North Europe). The region is defined by the resource group region used for creating the Arc cluster.
Ensure you meet the general prerequisites for cluster extensions, including the latest version of the k8s-extension Azure CLI extension.
cert-manager is required to support TLS for intracluster log communication. The examples later in this guide direct you though installation. For more information about cert-manager, see cert-manager.io

Install the Azure CLI and sign in, if you haven't already:

az login

Before you begin, set environment variables to be used for configuring Azure and cluster resources. If you already have a managed identity, Azure Key Vault, or other resource listed here, update the names in the environment variables to reflect those resources.

export RESOURCE_GROUP="AzureArcTest"
export CLUSTER_NAME="AzureArcTest1"
export LOCATION="EastUS"
export SUBSCRIPTION="$(az account show --query id --output tsv)"
az account set --subscription "${SUBSCRIPTION}"
export AZURE_TENANT_ID="$(az account show -s $SUBSCRIPTION --query tenantId --output tsv)"
export CURRENT_USER="$(az ad signed-in-user show --query userPrincipalName --output tsv)"
export KEYVAULT_NAME="my-kv"
export KEYVAULT_SECRET_NAME="my-secret"
export USER_ASSIGNED_IDENTITY_NAME="my-identity"
export FEDERATED_IDENTITY_CREDENTIAL_NAME="my-credential"
export KUBERNETES_NAMESPACE="my-namespace"
export SERVICE_ACCOUNT_NAME="my-service-account"

Activate workload identity federation in your cluster

The SSE uses a feature called workload identity federation to access and synchronize Azure Key Vault secrets. This section describes how to set this up. Following sections will explain how it is used in detail.

Tip

The following steps are based on the How-to guide for configuring Arc-enabled Kubernetes with workload identity federation. Refer to that documentation for any additional assistance.

If your cluster isn't yet connected to Azure Arc, follow these steps. During these steps, enable workload identity federation as part of the connect command:

az connectedk8s connect --name ${CLUSTER_NAME} --resource-group ${RESOURCE_GROUP} --enable-oidc-issuer --enable-workload-identity

If your cluster is already connected to Azure Arc, enable workload identity using the update command:

az connectedk8s update --name ${CLUSTER_NAME} --resource-group ${RESOURCE_GROUP} --enable-oidc-issuer --enable-workload-identity

Now configure your cluster to issue Service Account tokens with a new issuer URL (service-account-issuer) that enables Microsoft Entra ID to find the public keys necessary for it to validate these tokens. These public keys are for the cluster's own service account token issuer, and they were obtained and cloud-hosted at this URL as a result of the --enable-oidc-issuer option that you set above.

Optionally, you can also configure limits on the SSE's own permissions as a privileged resource running in the control plane by configuring OwnerReferencesPermissionEnforcement admission controller. This admission controller constrains how much the SSE can change other objects in the cluster.

Configure your kube-apiserver with the issuer URL field and permissions enforcement. The following example is for a k3s cluster. Your cluster may have different means for changing API server arguments: --kube-apiserver-arg="--service-account-issuer=${SERVICE_ACCOUNT_ISSUER}" and --kube-apiserver-arg="--enable-admission-plugins=OwnerReferencesPermissionEnforcement".
- Get the service account issuer URL.
```
export SERVICE_ACCOUNT_ISSUER="$(az connectedk8s show --name ${CLUSTER_NAME} --resource-group ${RESOURCE_GROUP} --query "oidcIssuerProfile.issuerUrl" --output tsv)"
echo $SERVICE_ACCOUNT_ISSUER
```
- Open the K3s server configuration file.
```
sudo nano /etc/systemd/system/k3s.service
```
- Edit the server configuration to look like the following example, replacing <SERVICE_ACCOUNT_ISSUER> with the above output from echo $SERVICE_ACCOUNT_ISSUER, remembering to include the trailing forward slash of this URL:
```
ExecStart=/usr/local/bin/k3s \
  server --write-kubeconfig-mode=644 \
     --kube-apiserver-arg="--service-account-issuer=<SERVICE_ACCOUNT_ISSUER>" \
     --kube-apiserver-arg="--enable-admission-plugins=OwnerReferencesPermissionEnforcement"
```

Restart your kube-apiserver.

sudo systemctl daemon-reload
sudo systemctl restart k3s

Use the How-to guide to activate workload identity federation on AKS on Azure Local by using the --enable-oidc-issuer and --enable-workload-identity flags.

Return to these steps after the initial activation. There is no need to complete the remainder of that guide.

Validate the activation has been succesful by obtaining the cluster's service account issuer URL. You'll use this URL in the following steps:

export SERVICE_ACCOUNT_ISSUER="$(az connectedk8s show --name ${CLUSTER_NAME} --resource-group ${RESOURCE_GROUP} --query "oidcIssuerProfile.issuerUrl" --output tsv)"
echo $SERVICE_ACCOUNT_ISSUER

Use the How-to guide to activate workload identity federation on AKS Edge Essentials.

Return to these steps after the initial activation. There is no need to complete the remainder of that guide.

Validate the activation has been succesful by obtaining the cluster's service account issuer URL. You'll use this URL in the following steps:

export SERVICE_ACCOUNT_ISSUER="$(az connectedk8s show --name ${CLUSTER_NAME} --resource-group ${RESOURCE_GROUP} --query "oidcIssuerProfile.issuerUrl" --output tsv)"
echo $SERVICE_ACCOUNT_ISSUER

Create a secret and configure an identity to access it

To access and synchronize a given Azure Key Vault secret, the SSE requires access to an Azure managed identity with appropriate Azure permissions to access that secret. The managed identity must be linked to a Kubernetes service account using the workload identity feature that you activated above. The SSE uses the associated federated Azure managed identity to pull secrets from Azure Key Vault to your Kubernetes secret store. The following sections describe how to set this up.

Create an Azure Key Vault

Create an Azure Key Vault and add a secret. If you already have an Azure Key Vault and secret, you can skip this section.

Create an Azure Key Vault:

az keyvault create --resource-group "${RESOURCE_GROUP}" --location "${LOCATION}" --name "${KEYVAULT_NAME}" --enable-rbac-authorization

Give yourself 'Secrets Officer' permissions on the vault, so you can create a secret:

az role assignment create --role "Key Vault Secrets Officer" --assignee ${CURRENT_USER} --scope /subscriptions/${SUBSCRIPTION}/resourcegroups/${RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/${KEYVAULT_NAME}

Create a secret and update it so you have two versions:

az keyvault secret set --vault-name "${KEYVAULT_NAME}" --name "${KEYVAULT_SECRET_NAME}" --value 'Hello!'
az keyvault secret set --vault-name "${KEYVAULT_NAME}" --name "${KEYVAULT_SECRET_NAME}" --value 'Hello2'

Create a user-assigned managed identity

Next, create a user-assigned managed identity and give it permissions to access the Azure Key Vault. If you already have a managed identity with Key Vault Reader and Key Vault Secrets User permissions to the Azure Key Vault, you can skip this section. For more information, see Create a user-assigned managed identities and Using Azure RBAC secret, key, and certificate permissions with Key Vault.

Create the user-assigned managed identity:

az identity create --name "${USER_ASSIGNED_IDENTITY_NAME}" --resource-group "${RESOURCE_GROUP}" --location "${LOCATION}" --subscription "${SUBSCRIPTION}"

Give the identity Key Vault Reader and Key Vault Secrets User permissions. You may need to wait a moment for replication of the identity creation before these commands succeed:

export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'clientId' -otsv)"
az role assignment create --role "Key Vault Reader" --assignee "${USER_ASSIGNED_CLIENT_ID}" --scope /subscriptions/${SUBSCRIPTION}/resourcegroups/${RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/${KEYVAULT_NAME}
az role assignment create --role "Key Vault Secrets User" --assignee "${USER_ASSIGNED_CLIENT_ID}" --scope /subscriptions/${SUBSCRIPTION}/resourcegroups/${RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/${KEYVAULT_NAME}

Create a federated identity credential

Create a Kubernetes service account for the workload that needs access to secrets. Then, create a federated identity credential to link between the managed identity, the OIDC service account issuer, and the Kubernetes Service Account.

Create a Kubernetes Service Account that will be federated to the managed identity. Annotate it with details of the associated user-assigned managed identity.

kubectl create ns ${KUBERNETES_NAMESPACE}

cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: ${SERVICE_ACCOUNT_NAME}
    namespace: ${KUBERNETES_NAMESPACE}
EOF

Create a federated identity credential:

az identity federated-credential create --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} --identity-name ${USER_ASSIGNED_IDENTITY_NAME} --resource-group ${RESOURCE_GROUP} --issuer ${SERVICE_ACCOUNT_ISSUER} --subject system:serviceaccount:${KUBERNETES_NAMESPACE}:${SERVICE_ACCOUNT_NAME}

Install the SSE

The SSE is available as an Azure Arc extension. An Azure Arc-enabled Kubernetes cluster can be extended with Azure Arc-enabled Kubernetes extensions. Extensions enable Azure capabilities on your connected cluster and provide an Azure Resource Manager-driven experience for the extension installation and lifecycle management.

cert-manager and trust-manager are also required for secure communication of logs between cluster services and must be installed before the Arc extension.

Install cert-manager.

helm repo add jetstack https://charts.jetstack.io/ --force-update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.16.2 --set crds.enabled=true

Install trust-manager.

helm upgrade trust-manager jetstack/trust-manager --install --namespace cert-manager --wait

Install the SSE to your Arc-enabled cluster using the following command:
```
az k8s-extension create \
  --cluster-name ${CLUSTER_NAME} \
  --cluster-type connectedClusters \
  --extension-type microsoft.azure.secretstore \
  --resource-group ${RESOURCE_GROUP} \
  --release-train preview \
  --name ssarcextension \
  --scope cluster 
```
If desired, you can optionally modify the default rotation poll interval by adding --configuration-settings rotationPollIntervalInSeconds=<time_in_seconds>:

Parameter name Description Default value

rotationPollIntervalInSeconds Specifies how quickly the SSE checks or updates the secret it's managing. 3600 (1 hour)

Parameter name	Description	Default value
`rotationPollIntervalInSeconds`	Specifies how quickly the SSE checks or updates the secret it's managing.	`3600` (1 hour)

Configure the SSE

Configure the installed extension with information about your Azure Key Vault and which secrets to synchronize to your cluster by defining instances of Kubernetes custom resources. You create two types of custom resources:

A SecretProviderClass object to define the connection to the Key Vault.
A SecretSync object for each secret to be synchronized.

Create a `SecretProviderClass` resource

The SecretProviderClass resource is used to define the connection to the Azure Key Vault, the identity to use to access the vault, which secrets to synchronize, and the number of versions of each secret to retain locally.

You need a separate SecretProviderClass for each Azure Key Vault you intend to synchronize, for each identity used for access to an Azure Key Vault, and for each target Kubernetes namespace.

Create one or more SecretProviderClass YAML files with the appropriate values for your Key Vault and secrets by following this example.

cat <<EOF > spc.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: secret-provider-class-name                      # Name of the class; must be unique per Kubernetes namespace
  namespace: ${KUBERNETES_NAMESPACE}                    # Kubernetes namespace to make the secrets accessible in
spec:
  provider: azure
  parameters:
    clientID: "${USER_ASSIGNED_CLIENT_ID}"               # Managed Identity Client ID for accessing the Azure Key Vault with.
    keyvaultName: ${KEYVAULT_NAME}                       # The name of the Azure Key Vault to synchronize secrets from.
    objects: |
      array:
        - |
          objectName: ${KEYVAULT_SECRET_NAME}            # The name of the secret to sychronize.
          objectType: secret
          objectVersionHistory: 2                       # [optional] The number of versions to synchronize, starting from latest.
    tenantID: "${AZURE_TENANT_ID}"                       # The tenant ID of the Key Vault 
EOF

Create a `SecretSync` object

Each synchronized secret also requires a SecretSync object, to define cluster-specific information. Here you specify information such as the name of the secret in your cluster and names for each version of the secret stored in your cluster.

Create one SecretSync object YAML file for each secret, following this template. The Kubernetes namespace should match the namespace of the matching SecretProviderClass.

cat <<EOF > ss.yaml
apiVersion: secret-sync.x-k8s.io/v1alpha1
kind: SecretSync
metadata:
  name: secret-sync-name                                  # Name of the object; must be unique per Kubernetes namespace
  namespace: ${KUBERNETES_NAMESPACE}                      # Kubernetes namespace
spec:
  serviceAccountName: ${SERVICE_ACCOUNT_NAME}             # The Kubernetes service account to be given permissions to access the secret.
  secretProviderClassName: secret-provider-class-name     # The name of the matching SecretProviderClass with the configuration to access the AKV storing this secret
  secretObject:
    type: Opaque
    data:
    - sourcePath: ${KEYVAULT_SECRET_NAME}/0                # Name of the secret in Azure Key Vault with an optional version number (defaults to latest)
      targetKey: ${KEYVAULT_SECRET_NAME}-data-key0         # Target name of the secret in the Kubernetes secret store (must be unique)
    - sourcePath: ${KEYVAULT_SECRET_NAME}/1                # [optional] Next version of the AKV secret. Note that versions of the secret must match the configured objectVersionHistory in the secrets provider class 
      targetKey: ${KEYVAULT_SECRET_NAME}-data-key1         # [optional] Next target name of the secret in the K8s secret store
EOF

Apply the configuration CRs

Apply the configuration custom resources (CRs) using the kubectl apply command:

kubectl apply -f ./spc.yaml
kubectl apply -f ./ss.yaml

The SSE automatically looks for the secrets and begins syncing them to the cluster.

View configuration options

To view additional configuration options for these two custom resource types, use the kubectl describe command to inspect the CRDs in the cluster:

# Get the name of any applied CRD(s)
kubectl get crds -o custom-columns=NAME:.metadata.name

# View the full configuration options and field parameters for a given CRD
kubectl describe crd secretproviderclass
kubectl describe crd secretsync

Observe secrets synchronizing to the cluster

Once the configuration is applied, secrets begin syncing to the cluster automatically at the cadence specified when installing the SSE.

View synchronized secrets

View the secrets synchronized to the cluster by running the following command:

# View a list of all secrets in the namespace
kubectl get secrets -n ${KUBERNETES_NAMESPACE}

# View details of all secrets in the namespace
kubectl get secrets -n ${KUBERNETES_NAMESPACE} -o yaml

View last sync status

To view the status of the most recent synchronization for a given secret, use the kubectl describe command for the SecretSync object. The output includes the secret creation timestamp, the versions of the secret, and detailed status messages for each synchronization event. This output can be used to diagnose connection or configuration errors, and to observe when the secret value changes.

kubectl describe secretsync secret-sync-name -n ${KUBERNETES_NAMESPACE}

View secrets values

To view the synchronized secret values, now stored in the Kubernetes secret store, use the following command:

kubectl get secret secret-sync-name -n ${KUBERNETES_NAMESPACE} -o jsonpath="{.data.${KEYVAULT_SECRET_NAME}-data-key0}" | base64 -d
kubectl get secret secret-sync-name -n ${KUBERNETES_NAMESPACE} -o jsonpath="{.data.${KEYVAULT_SECRET_NAME}-data-key1}" | base64 -d

Troubleshooting

The SSE is a Kubernetes deployment that contains a pod with two containers: the controller, which manages storing secrets in the cluster, and the provider, which manages access to, and pulling secrets from, the Azure Key Vault. Each synchronized secret has a SecretSync object that contains the status of the synchronization of that secret from Azure Key Vault to the cluster secret store.

To troubleshoot an issue, start by looking at the state of the SecretSync object, as described in View last sync status. The following table lists common status types, their meanings, and potential troubleshooting steps to resolve errors.

SecretSync Status Type	Details	Steps to fix/investigate further
`CreateSucceeded`	The secret was created successfully.	n/a
`CreateFailedProviderError`	Secret creation failed due to some issue with the provider (connection to Azure Key Vault). This failure could be due to internet connectivity, insufficient permissions for the identity syncing secrets, misconfiguration of the `SecretProviderClass`, or other issues.	Investigate further by looking at the logs of the provider using the following commands: `kubectl get pods -n azure-secret-store` `kubectl logs <secret-sync-controller-pod-name> -n azure-secret-store --container='provider-azure-installer'`
`CreateFailedInvalidLabel`	The secret creation failed because the secret already exists without the correct Kubernetes label that the SSE uses to manage its secrets.	Remove the existing label and secret and allow the SSE to recreate the secret: `kubectl delete secret <secret-name>` To force the SSE to recreate the secret faster than the configured rotation poll interval, delete the `SecretSync` object (`kubectl delete secretsync <secret-name>`) and reapply the secret sync class (`kubectl apply -f <path_to_secret_sync>`).
`CreateFailedInvalidAnnotation`	Secret creation failed because the secret already exists without the correct Kubernetes annotation that the SSE uses to manage its secrets.	Remove the existing annotation and secret and allow the SSE to recreate the secret: `kubectl delete secret <secret-name>` To force the SSE to recreate the secret faster than the configured rotation poll interval, delete the `SecretSync` object (`kubectl delete secretsync <secret-name>`) and reapply the secret sync class (`kubectl apply -f <path_to_secret_sync>`).
`UpdateNoValueChangeSucceeded`	The SSE checked Azure Key Vault for updates at the end of the configured poll interval, but there were no changes to sync.	n/a
`UpdateValueChangeOrForceUpdateSucceeded`	The SSE checked Azure Key Vault for updates and successfully updated the value.	n/a
`UpdateFailedInvalidLabel`	Secret update failed because the label on the secret that the SSE uses to manage its secrets was modified.	Remove the existing label and secret, and allow the SSE to recreate the secret: `kubectl delete secret <secret-name>` To force the SSE to recreate the secret faster than the configured rotation poll interval, delete the `SecretSync` object (`kubectl delete secretsync <secret-name>`) and reapply the secret sync class (`kubectl apply -f <path_to_secret_sync>`).
`UpdateFailedInvalidAnnotation`	Secret update failed because the annotation on the secret that the SSE uses to manage its secrets was modified.	Remove the existing annotation and secret and allow the SSE to recreate the secret: `kubectl delete secret <secret-name>` To force the SSE to recreate the secret faster than the configured rotation poll interval, delete the `SecretSync` object (`kubectl delete secretsync <secret-name>`) and reapply the secret sync class (`kubectl apply -f <path_to_secret_sync>`).
`UpdateFailedProviderError`	Secret update failed due to some issue with the provider (connection to Azure Key Vault). This failure could be due to internet connectivity, insufficient permissions for the identity syncing secrets, configuration of the `SecretProviderClass`, or other issues.	Investigate further by looking at the logs of the provider using the following commands: `kubectl get pods -n azure-secret-store` `kubectl logs <secret-sync-controller-pod-name> -n azure-secret-store --container='provider-azure-installer'`
`UserInputValidationFailed`	Secret update failed because the secret sync class was configured incorrectly (such as an invalid secret type).	Review the secret sync class definition and correct any errors. Then, delete the `SecretSync` object (`kubectl delete secretsync <secret-name>`), delete the secret sync class (`kubectl delete -f <path_to_secret_sync>`), and reapply the secret sync class (`kubectl apply -f <path_to_secret_sync>`).
`ControllerSpcError`	Secret update failed because the SSE failed to get the provider class or the provider class is misconfigured.	Review the provider class and correct any errors. Then, delete the `SecretSync` object (`kubectl delete secretsync <secret-name>`), delete the provider class (`kubectl delete -f <path_to_provider>`), and reapply the provider class (`kubectl apply -f <path_to_provider>`).
`ControllerInternalError`	Secret update failed due to an internal error in the SSE.	Check the SSE logs or the events for more information: `kubectl get pods -n azure-secret-store` `kubectl logs <secret-sync-controller-pod-name> -n azure-secret-store --container='manager'`
`SecretPatchFailedUnknownError`	Secret update failed during patching the Kubernetes secret value. This failure might occur if the secret was modified by someone other than the SSE or if there were issues during an update of the SSE.	Try deleting the secret and `SecretSync` object, then let the SSE recreate the secret by reapplying the secret sync CR: `kubectl delete secret <secret-name>` `kubectl delete secretsync <secret-name>` `kubectl apply -f <path_to_secret_sync>`

Remove the SSE

To remove the SSE and stop synchronizing secrets, uninstall it with the az k8s-extension delete command:

az k8s-extension delete --name ssarcextension --cluster-name $CLUSTER_NAME  --resource-group $RESOURCE_GROUP  --cluster-type connectedClusters

Uninstalling the extension doesn't remove secrets, SecretSync objects, or CRDs from the cluster. These objects must be removed directly with kubectl.

Deleting the SecretSync CRD removes all SecretSync objects, and by default removes all owned secrets, but secrets may persist if:

You modified ownership of any of the secrets.
You changed the garbage collection settings in your cluster, including setting different finalizers.

In the above cases, secrets must be deleted directly using kubectl.

Next steps

Learn more about Azure Arc extensions.
Learn more about Azure Key Vault.

Share via

Use the Secret Store extension to fetch secrets for offline access in Azure Arc-enabled Kubernetes clusters

Prerequisites

Activate workload identity federation in your cluster

Create a secret and configure an identity to access it

Create an Azure Key Vault

Create a user-assigned managed identity

Create a federated identity credential

Install the SSE

Configure the SSE

Create a `SecretProviderClass` resource

Create a `SecretSync` object

Apply the configuration CRs

View configuration options

Observe secrets synchronizing to the cluster

View synchronized secrets

View last sync status

View secrets values

Troubleshooting

Remove the SSE

Next steps

Feedback

Additional resources

Share via

Use the Secret Store extension to fetch secrets for offline access in Azure Arc-enabled Kubernetes clusters

Prerequisites

Activate workload identity federation in your cluster

Create a secret and configure an identity to access it

Create an Azure Key Vault

Create a user-assigned managed identity

Create a federated identity credential

Install the SSE

Configure the SSE

Create a SecretProviderClass resource

Create a SecretSync object

Apply the configuration CRs

View configuration options

Observe secrets synchronizing to the cluster

View synchronized secrets

View last sync status

View secrets values

Troubleshooting

Remove the SSE

Next steps

Feedback

Additional resources

Create a `SecretProviderClass` resource

Create a `SecretSync` object