ImagePullBackOff kubelet Failed to pull image .westeurope.data.azurecr.io : failed to resolve reference : failed to do request : tls: failed to verify certificate: x509: certificate is valid for *.azurecr.io, not .westeurope.data.azurecr.io

Question

Hi,

I 've got an issue on my new AKS cluster to pull image from my ACR :

• it seems to be great created:
• networking is good (can login/pull/push to the acr from a podman pod in the cluster),
• managedidentity with push/pull is role well created (az aks check-acr ... : "Your cluster can pull images from acrblablabla.azurecr.io")
  • but : when I create a pod , node can get the image from the acr

Name:         mypodubi8
Namespace:    default
Priority:     0
Node:         aks-.../x.x.x.x
Start Time:   Thu, 26 Sep 2024 14:47:15 +0000
Labels:       <none>
Annotations:  cni.projectcalico.org/containerID: a662e7ce540900006173f76ad65a7e16c6701fa86bfaf1a05c1d116dae4875e4
              cni.projectcalico.org/podIP: 172.25.5.24/32
              cni.projectcalico.org/podIPs: 172.25.5.24/32
Status:       Pending
IP:           172.25.5.24
IPs:
  IP:  172.25.5.24
Containers:
  mypodubi8:
    Container ID:  
    Image:         acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest
    Image ID:           Port:          <none>
    Host Port:     <none>
    Args:
      sleep
      1000000
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Environment:
      HTTP_PROXY:   http://proxy-cloud.xxx.cloud:80/
      http_proxy:   http://proxy-cloud.xxx.cloud:80/
      HTTPS_PROXY:  http://proxy-cloud.xxx.cloud:80/ 
      https_proxy:  http://proxy-cloud.xxx.cloud:80/
      NO_PROXY:     blablabla,acrblablabla.westeurope.data.azurecr.io,acrblablabla.azurecr.io 
      no_proxy:     blablabla,acrblablabla.westeurope.data.azurecr.io,acrblablabla.azurecr.io 
      Mounts:       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lkfkj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-lkfkj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  43s                default-scheduler  Successfully assigned default/mypodubi8 to aks-xxx
  Normal   Pulling    27s (x2 over 43s)  kubelet            Pulling image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest" 
  Warning  Failed     27s (x2 over 42s)  kubelet            Failed to pull image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest": failed to pull and unpack image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest": failed to resolve reference "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest": failed to do request: Head "https://acrblablabla.westeurope.data.azurecr.io/v2/v
alid/ubi8/manifests/latest": tls: failed to verify certificate: x509: certificate is valid for *.azurecr.io, not acrblablabla.westeurop e.data.azurecr.io 
  Warning  Failed     27s (x2 over 42s)  kubelet            Error: ErrImagePull 
  Normal   BackOff    12s (x3 over 42s)  kubelet            Back-off pulling image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8: latest" 
  Warning  Failed     12s (x3 over 42s)  kubelet            Error: ImagePullBackOff

Please, help is needed,
Thomas

Answer 1

Hi Thomas FOUREST (admintenant.tfo),
Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

An ImagePullBackOff is the result of repeat ErrImagePull errors, meaning the kubelet tried to pull a container image several times and failed. This indicates a persistent problem that needs to be addressed.
The issue you're experiencing here is due to a certificate verification error when trying to pull an image from your Azure Container Registry (ACR). The error message indicates that the certificate is valid for *.azurecr.io, but not for acrblablabla.westeurope.data.azurecr.io.

I can provide you the below document which matches with your scenario:-)
https://github.com/Azure/acr/issues/425

If you have any further queries, do let us know.

If the answer is helpful, please click "Accept Answer" and "Upvote it."

Answer 2

Hello,

First of all, Thanks for the support !

We solve the issue with new entries in our private DNS zone for a best resolution of the private endpoint that link the ACR PaaS in our subnet.
So the certificat error will be the good symptom to find the root cause.

Best regards,

Thomas.

Answer 3

Hi Thomas FOUREST (admintenant.tfo),

Hope you are doing good!
Glad to know that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Issue: The AKS cluster is unable to pull the image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest" from your ACR. The error is due to a TLS certificate verification failure. The certificate is valid for "*.azurecr.io", but not for "acrblablabla.westeurope.data.azurecr.io".

Solution: Created new entries in their private DNS zone, which helps to properly resolve the private endpoint linking the Azure Container Registry (ACR) PaaS in their subnet. The certificate error is a good indication of the root cause of the problem.

If you have any other questions or are still running into more issues, please let me know. Thank you again for your time and patience throughout this issue.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Thank you!