Compartilhar via

Configurar e implantar um cluster do Valkey no AKS (Serviço de Kubernetes do Azure)

  • Artigo

Neste artigo, vamos configurar e implantar um cluster do Valkey no AKS (Serviço de Kubernetes do Azure).

Observação

Este artigo contém referências aos termos mestre e subordinado, que são termos que a Microsoft não usa mais. Quando os termos forem removidos do software Valkey, também vamos removê-los deste artigo.

Criar um namespace

  1. Crie um namespace para o cluster do Valkey usando o comando kubectl create namespace.

    kubectl create namespace ${SERVICE_ACCOUNT_NAMESPACE} --dry-run=client --output yaml | kubectl apply -f -

    Exemplo de saída:

    namespace/valkey created

Criar segredos

  1. Gerar uma senha aleatória para o cluster Valkey usando openssl e armazená-la no seu Azure Key Vault usando o comando az keyvault secret set. Defina a política para permitir que a identidade atribuída pelo usuário obtenha o segredo usando o comando az keyvault set-policy.

    SECRET=$(openssl rand -base64 32)
echo requirepass $SECRET > /tmp/valkey-password-file.conf
echo primaryauth $SECRET >> /tmp/valkey-password-file.conf
az keyvault secret set --vault-name $MY_KEYVAULT_NAME --name valkey-password-file --file /tmp/valkey-password-file.conf --output table
rm /tmp/valkey-password-file.conf
az keyvault set-policy --name $MY_KEYVAULT_NAME --object-id $userAssignedObjectID --secret-permissions get --output table

  2. Crie um recurso de SecretProviderClass para acessar a senha do Valkey armazenada no cofre de chaves usando o comando kubectl apply.

    kubectl apply -f - <<EOF
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: valkey-password
  namespace: valkey
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "${userAssignedIdentityID}"
    keyvaultName: ${MY_KEYVAULT_NAME}              # the name of the AKV instance
    objects: |
      array:
        - |
          objectName: valkey-password-file
          objectAlias: valkey-password-file.conf
          objectType: secret
    tenantId: "${TENANT_ID}" # the tenant ID of the AKV instance
EOF

Implantar o cluster do Valkey

  1. Crie um ConfigMap montado como um volume no StatefulSet do Valkey a ser usado para configurar o cluster do Valkey usando o comando kubectl apply.

    kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: valkey-cluster
  namespace: valkey
data:
  valkey.conf:  |+
    cluster-enabled yes
    cluster-node-timeout 15000
    cluster-config-file /data/nodes.conf
    appendonly yes
    protected-mode yes
    dir /data
    port 6379
    include /etc/valkey-password/valkey-password-file.conf
EOF

    Exemplo de saída:

    configmap/valkey-cluster created

  2. Criar um recurso de StatefulSet com uma meta spec.affinity para manter todos os primários na zona 2, preferencialmente em nós diferentes, usando o comando kubectl apply.

    kubectl apply -f - <<EOF
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: valkey-masters
  namespace: valkey
spec:
  serviceName: "valkey-masters"
  replicas: 3
  selector:
    matchLabels:
      app: valkey
  template:
    metadata:
      labels:
        app: valkey
        appCluster: valkey-masters
    spec:
      terminationGracePeriodSeconds: 20
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: agentpool
                operator: In
                values:
                - valkey
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - ${MY_LOCATION}-1
            - matchExpressions:
              - key: agentpool
                operator: In
                values:
                - valkey
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - ${MY_LOCATION}-2
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 90
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - valkey
              topologyKey: topology.kubernetes.io/zone
          - weight: 90
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - valkey
              topologyKey: kubernetes.io/hostname
      containers:
      - name: valkey
        image: "${MY_ACR_REGISTRY}.azurecr.io/valkey:latest"
        env:
        - name: VALKEY_PASSWORD_FILE
          value: "/etc/valkey-password/valkey-password-file.conf"
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        command:
          - "valkey-server"
        args:
          - "/conf/valkey.conf"
          - "--cluster-announce-ip"
          - "\$(MY_POD_IP)"
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
        ports:
            - name: valkey
              containerPort: 6379
              protocol: "TCP"
            - name: cluster
              containerPort: 16379
              protocol: "TCP"
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
        - name: valkey-password
          mountPath: /etc/valkey-password
          readOnly: true
      volumes:
      - name: valkey-password
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: valkey-password
      - name: conf
        configMap:
          name: valkey-cluster
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: managed-csi-premium
      resources:
        requests:
          storage: 20Gi
EOF

    Exemplo de saída:

    statefulset.apps/valkey-masters created

  3. Crie um segundo recurso StatefulSet para os secundários do Valkey com uma meta spec.affinity para manter todas as réplicas na zona 3, preferencialmente em nós diferentes, usando o comando kubectl apply.

    kubectl apply -f - <<EOF
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: valkey-replicas
  namespace: valkey
spec:
  serviceName: "valkey-replicas"
  replicas: 3
  selector:
    matchLabels:
      app: valkey
  template:
    metadata:
      labels:
        app: valkey
        appCluster: valkey-replicas
    spec:
      terminationGracePeriodSeconds: 20
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: agentpool
                operator: In
                values:
                - valkey
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - ${MY_LOCATION}-3
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 90
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - valkey
              topologyKey: kubernetes.io/hostname
      containers:
      - name: valkey
        image: "${MY_ACR_REGISTRY}.azurecr.io/valkey:latest"
        env:
        - name: VALKEY_PASSWORD_FILE
          value: "/etc/valkey-password/valkey-password-file.conf"
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        command:
          - "valkey-server"
        args:
          - "/conf/valkey.conf"
          - "--cluster-announce-ip"
          - "\$(MY_POD_IP)"
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
        ports:
            - name: valkey
              containerPort: 6379
              protocol: "TCP"
            - name: cluster
              containerPort: 16379
              protocol: "TCP"
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
        - name: valkey-password
          mountPath: /etc/valkey-password
          readOnly: true
      volumes:
      - name: valkey-password
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: valkey-password
      - name: conf
        configMap:
          name: valkey-cluster
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: managed-csi-premium
      resources:
        requests:
          storage: 20Gi
EOF

    Exemplo de saída:

    statefulset.apps/valkey-replicas created

  4. Verifique se master-N e replica-N estão em execução em diferentes nós e zonas usando os comandos kubectl get nodes e kubectl get pods.

    kubectl get pods -n valkey -o wide
kubectl get node -o custom-columns=Name:.metadata.name,Zone:".metadata.labels.topology\.kubernetes\.io/zone"

    Exemplo de saída:

    NAME                READY   STATUS    RESTARTS   AGE     IP             NODE                             NOMINATED NODE   READINESS GATES
valkey-masters-0    1/1     Running   0          2m55s   10.224.0.4     aks-valkey-18693609-vmss000004   <none>           <none>
valkey-masters-1    1/1     Running   0          2m31s   10.224.0.137   aks-valkey-18693609-vmss000000   <none>           <none>
valkey-masters-2    1/1     Running   0          2m7s    10.224.0.222   aks-valkey-18693609-vmss000001   <none>           <none>
valkey-replicas-0   1/1     Running   0          88s     10.224.0.237   aks-valkey-18693609-vmss000005   <none>           <none>
valkey-replicas-1   1/1     Running   0          70s     10.224.0.18    aks-valkey-18693609-vmss000002   <none>           <none>
valkey-replicas-2   1/1     Running   0          48s     10.224.0.242   aks-valkey-18693609-vmss000005   <none>           <none>
Name                                Zone
aks-nodepool1-17621399-vmss000000   centralus-1
aks-nodepool1-17621399-vmss000001   centralus-2
aks-nodepool1-17621399-vmss000003   centralus-3
aks-valkey-18693609-vmss000000      centralus-1
aks-valkey-18693609-vmss000001      centralus-2
aks-valkey-18693609-vmss000002      centralus-3
aks-valkey-18693609-vmss000003      centralus-1
aks-valkey-18693609-vmss000004      centralus-2
aks-valkey-18693609-vmss000005      centralus-3

    Espere que todos os pods estejam em execução antes de prosseguir para a próxima etapa.

  5. Crie três recursos de Service headless (o primeiro para todo o cluster, o segundo para os primários e o terceiro para os secundários) a serem usados para obter os endereços IP dos pods do Valkey usando o comando kubectl apply.

    kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: valkey-cluster
  namespace: valkey
spec:
  clusterIP: None
  ports:
  - name: valkey-port
    port: 6379
    protocol: TCP
    targetPort: 6379
  selector:
    app: valkey
  sessionAffinity: None
  type: ClusterIP
EOF

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: valkey-masters
  namespace: valkey
spec:
  clusterIP: None
  ports:
  - name: valkey-port
    port: 6379
    protocol: TCP
    targetPort: 6379
  selector:
    app: valkey
    appCluster: valkey-masters
  sessionAffinity: None
  type: ClusterIP
EOF

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: valkey-replicas
  namespace: valkey
spec:
  clusterIP: None
  ports:
  - name: valkey-port
    port: 6379
    protocol: TCP
    targetPort: 6379
  selector:
    app: valkey
    appCluster: valkey-replicas
  sessionAffinity: None
  type: ClusterIP
EOF

    Exemplo de saída:

    service/valkey-cluster created
service/valkey-masters created
service/valkey-replicas created

Executar o cluster do Valkey

  1. Adicione as primárias do Valkey, na zona 1 e 2, ao cluster usando o comando kubectl exec.

    kubectl exec -it -n valkey valkey-masters-0 -- valkey-cli --cluster create --cluster-yes --cluster-replicas 0 \
                    valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379 \
                    valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379 \
                    valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379 \
                    --pass ${SECRET}

    Exemplo de saída:

    >>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379
   slots:[0-5460] (5461 slots) master
M: fd1fb98db83976478e05edd3d2a02f9a13badd80 valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379
   slots:[5461-10922] (5462 slots) master
M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379
   slots:[10923-16383] (5461 slots) master
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379)
M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379
   slots:[0-5460] (5461 slots) master
M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e 10.224.0.176:6379
   slots:[10923-16383] (5461 slots) master
M: fd1fb98db83976478e05edd3d2a02f9a13badd80 10.224.0.247:6379
   slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

  2. Adicione as réplicas Valkey, na zona 3, ao cluster usando o comando kubectl exec.

    kubectl exec -ti -n valkey valkey-masters-0 -- valkey-cli --cluster add-node \
                    valkey-replicas-0.valkey-replicas.valkey.svc.cluster.local:6379 \
                    valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379  --cluster-slave \
                    --pass ${SECRET}

kubectl exec -ti -n valkey valkey-masters-0 -- valkey-cli --cluster add-node \
                    valkey-replicas-1.valkey-replicas.valkey.svc.cluster.local:6379 \
                    valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379  --cluster-slave \
                    --pass ${SECRET}

kubectl exec -ti -n valkey valkey-masters-0 -- valkey-cli --cluster add-node \
                    valkey-replicas-2.valkey-replicas.valkey.svc.cluster.local:6379 \
                    valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379  --cluster-slave \
                    --pass ${SECRET}

    Exemplo de saída:

    >>> Adding node valkey-replicas-0.valkey-replicas.valkey.svc.cluster.local:6379 to cluster valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379
>>> Performing Cluster Check (using node valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379)
M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379
   slots:[0-5460] (5461 slots) master
M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e 10.224.0.176:6379
   slots:[10923-16383] (5461 slots) master
M: fd1fb98db83976478e05edd3d2a02f9a13badd80 10.224.0.247:6379
   slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379
>>> Send CLUSTER MEET to node valkey-replicas-0.valkey-replicas.valkey.svc.cluster.local:6379 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379.
[OK] New node added correctly.
>>> Adding node valkey-replicas-1.valkey-replicas.valkey.svc.cluster.local:6379 to cluster valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379
>>> Performing Cluster Check (using node valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379)
M: fd1fb98db83976478e05edd3d2a02f9a13badd80 valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379
   slots:[5461-10922] (5462 slots) master
S: 0ebceb60cbcc31da9040159440a1f4856b992907 10.224.0.224:6379
   slots: (0 slots) slave
   replicates ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35
M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e 10.224.0.176:6379
   slots:[10923-16383] (5461 slots) master
M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 10.224.0.14:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379
>>> Send CLUSTER MEET to node valkey-replicas-1.valkey-replicas.valkey.svc.cluster.local:6379 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379.
[OK] New node added correctly.
>>> Adding node valkey-replicas-2.valkey-replicas.valkey.svc.cluster.local:6379 to cluster valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379
>>> Performing Cluster Check (using node valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379)
M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379
   slots:[10923-16383] (5461 slots) master
S: 0ebceb60cbcc31da9040159440a1f4856b992907 10.224.0.224:6379
   slots: (0 slots) slave
   replicates ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35
S: fa44edff683e2e01ee5c87233f9f3bc35c205dce 10.224.0.103:6379
   slots: (0 slots) slave
   replicates fd1fb98db83976478e05edd3d2a02f9a13badd80
M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 10.224.0.14:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: fd1fb98db83976478e05edd3d2a02f9a13badd80 10.224.0.247:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379
>>> Send CLUSTER MEET to node valkey-replicas-2.valkey-replicas.valkey.svc.cluster.local:6379 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379.
[OK] New node added correctly.

  3. Verifique as funções dos pods usando os seguintes comandos:

    for x in $(seq 0 2); do echo "valkey-masters-$x"; kubectl exec -n valkey valkey-masters-$x  -- valkey-cli --pass ${SECRET} role; echo; done
for x in $(seq 0 2); do echo "valkey-replicas-$x"; kubectl exec -n valkey valkey-replicas-$x -- valkey-cli --pass ${SECRET} role; echo; done

    Saída de exemplo:

    valkey-masters-0
master
84
10.224.0.224
6379
84

valkey-masters-1
master
84
10.224.0.103
6379
84

valkey-masters-2
master
70
10.224.0.200
6379
70

valkey-replicas-0
slave
10.224.0.14
6379
connected
98

valkey-replicas-1
slave
10.224.0.247
6379
connected
98

valkey-replicas-2
slave
10.224.0.176
6379
connected
84

Próximas etapas

Para saber mais sobre como implantar software de código aberto no AKS (Serviço de Kubernetes do Azure), consulte os seguintes artigos:

Colaboradores

A Microsoft atualiza este artigo. Os seguintes colaboradores escreveram originalmente:

  • Nelly Kiboi | Engenheiro de Serviços
  • Saverio Proto | Engenheiro de Experiência do Cliente Principal
  • Don High | Engenheiro de Cliente Principal
  • LaBrina Loving | Engenheira de Serviço principal
  • Ken Kilty | Principal TPM
  • Russell de Pina | Principal TPM
  • Colin Mixon | Gerente de Produto
  • Ketan Chawda | Engenheiro de Clientes Sênior
  • Naveed Kharadi | Engenheiro de Experiência do Cliente
  • Erin Schaffer | Desenvolvedora de Conteúdo 2