Integrate Azure HPC Cache with Azure Kubernetes Service (AKS)
Azure HPC Cache speeds access to your data for high-performance computing (HPC) tasks. By caching files in Azure, Azure HPC Cache brings the scalability of cloud computing to your existing workflow. This article shows you how to integrate Azure HPC Cache with Azure Kubernetes Service (AKS).
Before you begin
AKS cluster must be in a region that supports Azure HPC Cache.
You need Azure CLI version 2.7 or later. Run
az --version
to find the version. If you need to install or upgrade, see Install Azure CLI.Register the
hpc-cache
extension in your Azure subscription. For more information on using HPC Cache with Azure CLI, see the HPC Cache CLI prerequisites.Review the HPC Cache prerequisites. You need to satisfy the following before you can run an HPC Cache:
- The cache requires a dedicated subnet with at least 64 IP addresses available.
- The subnet must not host other VMs or containers.
- The subnet must be accessible from the AKS nodes.
If you need to run your application as a user without root access, you may need to disable root squashing by using the change owner (chown) command to change directory ownership to another user. The user without root access needs to own a directory to access the file system. For the user to own a directory, the root user must chown a directory to that user, but if the HPC Cache is squashing root, this operation is denied because the root user (UID 0) is being mapped to the anonymous user. For more information about root squashing and client access policies, see HPC Cache access policies.
Install the hpc-cache
Azure CLI extension
Important
AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:
To install the hpc-cache extension, run the following command:
az extension add --name hpc-cache
Run the following command to update to the latest version of the extension released:
az extension update --name hpc-cache
Register the StorageCache feature flag
Register the Microsoft.StorageCache resource provider using the az provider register
command.
az provider register --namespace Microsoft.StorageCache --wait
It takes a few minutes for the status to show Registered. Verify the registration status by using the az feature show command:
az feature show --namespace "Microsoft.StorageCache"
Create the Azure HPC Cache
Get the node resource group using the
az aks show
command with the--query nodeResourceGroup
query parameter.az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv
Your output should look similar to the following example output:
MC_myResourceGroup_myAKSCluster_eastus
Create a dedicated HPC Cache subnet using the
az network vnet subnet create
command. First define the environment variables forRESOURCE_GROUP
,VNET_NAME
,VNET_ID
, andSUBNET_NAME
. Copy the output from the previous step forRESOURCE_GROUP
, and specify a value forSUBNET_NAME
.RESOURCE_GROUP=MC_myResourceGroup_myAKSCluster_eastus VNET_NAME=$(az network vnet list --resource-group $RESOURCE_GROUP --query [].name -o tsv) VNET_ID=$(az network vnet show --resource-group $RESOURCE_GROUP --name $VNET_NAME --query "id" -o tsv) SUBNET_NAME=MyHpcCacheSubnet
az network vnet subnet create \ --resource-group $RESOURCE_GROUP \ --vnet-name $VNET_NAME \ --name $SUBNET_NAME \ --address-prefixes 10.0.0.0/26
Create an HPC Cache in the same node resource group and region. First define the environment variable
SUBNET_ID
.SUBNET_ID=$(az network vnet subnet show --resource-group $RESOURCE_GROUP --vnet-name $VNET_NAME --name $SUBNET_NAME --query "id" -o tsv)
Create the HPC Cache using the
az hpc-cache create
command. The following example creates the HPC Cache in the East US region with a Standard 2G cache type named MyHpcCache. Specify a value for --location, --sku-name, and --name.az hpc-cache create \ --resource-group $RESOURCE_GROUP \ --cache-size-gb "3072" \ --location eastus \ --subnet $SUBNET_ID \ --sku-name "Standard_2G" \ --name MyHpcCache
Note
Creation of the HPC Cache can take up to 20 minutes.
Create and configure Azure storage
Create a storage account using the
az storage account create
command. First define the environment variableSTORAGE_ACCOUNT_NAME
.Important
You need to select a unique storage account name. Replace
uniquestorageaccount
with your specified name. Storage account names must be between 3 and 24 characters in length and can contain only numbers and lowercase letters.STORAGE_ACCOUNT_NAME=uniquestorageaccount
The following example creates a storage account in the East US region with the Standard_LRS SKU. Specify a value for --location and --sku.
az storage account create \ --name $STORAGE_ACCOUNT_NAME \ --resource-group $RESOURCE_GROUP \ --location eastus \ --sku Standard_LRS
Assign the Storage Blob Data Contributor Role on your subscription using the
az role assignment create
command. First, define the environment variablesSTORAGE_ACCOUNT_ID
andAD_USER
.STORAGE_ACCOUNT_ID=$(az storage account show --name $STORAGE_ACCOUNT_NAME --query "id" -o tsv) AD_USER=$(az ad signed-in-user show --query objectId -o tsv)
az role assignment create --role "Storage Blob Data Contributor" --assignee $AD_USER --scope $STORAGE_ACCOUNT_ID
Create the Blob container within the storage account using the
az storage container create
command. First, define the environment variableCONTAINER_NAME
and replace the name for the Blob container.CONTAINER_NAME=mystoragecontainer
az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME --auth-mode login
Provide permissions to the Azure HPC Cache service account to access your storage account and Blob container using the
az role assignment
commands. First, define the environment variablesHPC_CACHE_USER
andHPC_CACHE_ID
.HPC_CACHE_USER="StorageCache Resource Provider" HPC_CACHE_ID=$(az ad sp list --display-name "${HPC_CACHE_USER}" --query "[].objectId" -o tsv)
az role assignment create --role "Storage Account Contributor" --assignee $HPC_CACHE_ID --scope $STORAGE_ACCOUNT_ID az role assignment create --role "Storage Blob Data Contributor" --assignee $HPC_CACHE_ID --scope $STORAGE_ACCOUNT_ID
Add the blob container to your HPC Cache as a storage target using the
az hpc-cache blob-storage-target add
command. The following example creates a blob container named MyStorageTarget to the HPC Cache MyHpcCache. Specify a value for --name, --cache-name, and --virtual-namespace-path.az hpc-cache blob-storage-target add \ --resource-group $RESOURCE_GROUP \ --cache-name MyHpcCache \ --name MyStorageTarget \ --storage-account $STORAGE_ACCOUNT_ID \ --container-name $CONTAINER_NAME \ --virtual-namespace-path "/myfilepath"
Set up client load balancing
Create an Azure Private DNS zone for the client-facing IP addresses using the
az network private-dns zone create
command. First define the environment variablePRIVATE_DNS_ZONE
and specify a name for the zone.PRIVATE_DNS_ZONE="myhpccache.local"
az network private-dns zone create \ --resource-group $RESOURCE_GROUP \ --name $PRIVATE_DNS_ZONE
Create a DNS link between the Azure Private DNS Zone and the VNet using the
az network private-dns link vnet create
command. Replace the value for --name.az network private-dns link vnet create \ --resource-group $RESOURCE_GROUP \ --name MyDNSLink \ --zone-name $PRIVATE_DNS_ZONE \ --virtual-network $VNET_NAME \ --registration-enabled true
Create the round-robin DNS name for the client-facing IP addresses using the
az network private-dns record-set a create
command. First, define the environment variablesDNS_NAME
,HPC_MOUNTS0
,HPC_MOUNTS1
, andHPC_MOUNTS2
. Replace the value for the propertyDNS_NAME
.DNS_NAME="server" HPC_MOUNTS0=$(az hpc-cache show --name "MyHpcCache" --resource-group $RESOURCE_GROUP --query "mountAddresses[0]" -o tsv | tr --delete '\r') HPC_MOUNTS1=$(az hpc-cache show --name "MyHpcCache" --resource-group $RESOURCE_GROUP --query "mountAddresses[1]" -o tsv | tr --delete '\r') HPC_MOUNTS2=$(az hpc-cache show --name "MyHpcCache" --resource-group $RESOURCE_GROUP --query "mountAddresses[2]" -o tsv | tr --delete '\r')
az network private-dns record-set a add-record -g $RESOURCE_GROUP -z $PRIVATE_DNS_ZONE -n $DNS_NAME -a $HPC_MOUNTS0 az network private-dns record-set a add-record -g $RESOURCE_GROUP -z $PRIVATE_DNS_ZONE -n $DNS_NAME -a $HPC_MOUNTS1 az network private-dns record-set a add-record -g $RESOURCE_GROUP -z $PRIVATE_DNS_ZONE -n $DNS_NAME -a $HPC_MOUNTS2
Create a persistent volume
Create a file named
pv-nfs.yaml
to define a persistent volume and then paste in the following manifest. Replace the values for the propertyserver
andpath
.--- apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs spec: capacity: storage: 10000Gi accessModes: - ReadWriteMany mountOptions: - vers=3 nfs: server: server.myhpccache.local path: /
Get the credentials for your Kubernetes cluster using the
az aks get-credentials
command.az aks get-credentials --resource-group myResourceGroup --name myAKSCluster
Create the persistent volume using the
kubectl apply
command.kubectl apply -f pv-nfs.yaml
Verify the status of the persistent volume is Available using the
kubectl describe
command.kubectl describe pv pv-nfs
Create the persistent volume claim
Create a file named
pvc-nfs.yaml
to define a persistent volume claim, and then paste the following manifest.apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-nfs spec: accessModes: - ReadWriteMany storageClassName: "" resources: requests: storage: 100Gi
Create the persistent volume claim using the
kubectl apply
command.kubectl apply -f pvc-nfs.yaml
Verify the status of the persistent volume claim is Bound using the
kubectl describe
command.kubectl describe pvc pvc-nfs
Mount the HPC Cache with a pod
Create a file named
nginx-nfs.yaml
to define a pod that uses the persistent volume claim, and then paste the following manifest.kind: Pod apiVersion: v1 metadata: name: nginx-nfs spec: containers: - image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine name: nginx-nfs command: - "/bin/sh" - "-c" - while true; do echo $(date) >> /mnt/azure/myfilepath/outfile; sleep 1; done volumeMounts: - name: disk01 mountPath: /mnt/azure volumes: - name: disk01 persistentVolumeClaim: claimName: pvc-nfs
Create the pod using the
kubectl apply
command.kubectl apply -f nginx-nfs.yaml
Verify the pod is running using the
kubectl describe
command.kubectl describe pod nginx-nfs
Verify your volume is mounted in the pod using the
kubectl exec
command to connect to the pod.kubectl exec -it nginx-nfs -- sh
To check if the volume is mounted, run
df
in its human-readable format using the--human-readable
(-h
for short) option.df -h
The following example resembles output returned from the command:
Filesystem Size Used Avail Use% Mounted on ... server.myhpccache.local:/myfilepath 8.0E 0 8.0E 0% /mnt/azure/myfilepath ...
Next steps
- For more information on Azure HPC Cache, see HPC Cache overview.
- For more information on using NFS with AKS, see Manually create and use a Network File System (NFS) Linux Server volume with AKS.
Azure Kubernetes Service