Volume Snapshots Fail on AKS with 502 and/or timeout or HTML error page

Jeff Mealo 0

Volume snapshots are failing repeatedly, sometimes with 502 or HTML-based error pages from Azure, here's an example:

  Warning  Error           6m24s  cloudnative-pg-backup  snapshot backup failed: Failed to create snapshot: failed to take snapshot of the volume /subscriptions/19ea338e-67f7-4a2f-a165-6ccdb9c1aecb/resourceGroups/mc_application_production-app-cluster_eastus/providers/Microsoft.Compute/disks/pvc-37f06647-f7f6-4838-bf05-2a2f1e054a63: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"

Jeff Mealo 0

  Normal  CreatingSnapshot  9m55s  snapshot-controller  Waiting for a snapshot db/production-pg-etl-hourly-volume-snapshot-20250213200000 to be created by the CSI driver.

Nagarjuna Reddy Yanna 90 Reputation points Microsoft Vendor

2025-02-14T12:35:58.08+00:00

Hi Jeff Mealo,

Thank you for reaching out to us on the Microsoft Q&A forum.

Could you please follow the below steps for you getting 502 bad gateway error.

Server Error: 502 - Web server received an invalid response while acting as a gateway or proxy server:

Please confirm whether it is required that the CNAME of the TLS certificate installed on the backend server, matches the host name configured in the HTTP backend settings, otherwise the second part of the End-to-end communication that happens between the instances of the Application Gateway and the backend, will fail with "Upstream SSL certificate does not match", and will throw back a Server Error: 502

Make sure to increase timeout request so that we can overcome the timeout issue.

Application Gateway allows you to configure this setting via the BackendHttpSetting, which can be then applied to different pools. Different backend pools can have different BackendHttpSetting, and a different request time-out configured.

"New-AzApplicationGatewayBackendHttpSettings -Name 'Setting01' -Port 80 -Protocol Http -CookieBasedAffinity Enabled -RequestTimeout 60"

For your reference, please find the below document.

https://docs.azure.cn/en-us/backup/azure-kubernetes-service-backup-troubleshoot

https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502

If the information is helpful, please consider by clicking the "Upvote" on the post.

If you have any further queries, please let us know in the comment.
Jeff Mealo 0 Reputation points

2025-02-14T14:24:05.3566667+00:00

@Nagarjuna Reddy Yanna Thanks for your response. It doesn't appear to be addressing the concerns in my post. Azure's volume snapshot service (your internal APIs) are returning the errors. Volume snapshotting is not working. We've turned on Diagnostic Settings for the AKS cluster and are collecting the relevant logs.
Nagarjuna Reddy Yanna 90 Reputation points Microsoft Vendor

2025-02-17T10:16:15.51+00:00

Hi Jeff Mealo,

Thank you for reaching us again and sharing the information.

Could you please check below blow points.

Examine the CSI Driver Logs: Look into the logs related to the Azure Disk CSI driver on your AKS nodes. Focus specifically on any errors or warnings related to snapshot operations.

Check for Supported Drivers:

Ensure that you're using a CSI driver that supports volume snapshots, such as disk.csi.azure.com or file.csi.azure.com.

For your reference, please find the below document.

https://learn.microsoft.com/en-us/azure/storage/container-storage/volume-snapshot-restore

https://learn.microsoft.com/en-us/azure/aks/azure-disk-csi

If the information is helpful, please consider by clicking the "Upvote" on the post.

If you have any further queries, please let us know in the comment.

Thank you.
Nagarjuna Reddy Yanna 90 Reputation points Microsoft Vendor

2025-02-18T09:09:43.8033333+00:00

Hi Jeff Mealo,

I wanted to check if you had the opportunity to review the information which was provided in my previous posted comment.

If it was helpful, please click "Upvote" on his post to let us know.

Thank You.
Jeff Mealo 0 Reputation points

2025-02-18T20:03:26.73+00:00

The Azure Disk CSI driver doesn't log anything:
❯ kubectl logs -n kube-system csi-azuredisk-node-vmgvn -f

Defaulted container "liveness-probe" out of: liveness-probe, node-driver-registrar, azuredisk

I0218 16:23:12.553420 1 main.go:135] "Calling CSI driver to discover driver name"

I0218 16:23:12.557777 1 main.go:143] "CSI driver name" driver="disk.csi.azure.com"

I0218 16:23:12.557804 1 main.go:172] "ServeMux listening" address="0.0.0.0:29603"
Jeff Mealo 0 Reputation points

2025-02-18T20:04:41.7666667+00:00

Could someone consult with an engineer and not an LLM? Please escalate.
Markapuram Sudheer Reddy 990 Reputation points Microsoft Vendor

2025-02-19T20:15:37.4866667+00:00

Hi Jeff Mealo,

I have provided you the workarounds in the PM regarding the issue you have mentioned in the above.

Let me know, if you have any queries, we will assist you.

Thank You.
Jeff Mealo 0 Reputation points

2025-02-19T21:24:34.76+00:00

I see that you sent me an empty message?
Jeff Mealo 0 Reputation points

2025-02-19T21:27:51.5533333+00:00

The PM feature is broken. You have to inspect the network activity to even get the message contents.
Jeff Mealo 0 Reputation points

2025-02-19T21:32:14.0966667+00:00
You sent my co-worker the same canned response.

Connectivity to blob storage from our AKS cluster has nothing to do with volume snapshots failing. (We do not have connectivity issues to blob storage and use it heavily)

There is not a configuration issue with the CSI driver or VolumeSnapshotClass as they work sometimes and not others.
Jeff Mealo 0 Reputation points

2025-02-19T21:32:36.99+00:00
You sent my co-worker the same canned response.

Connectivity to blob storage from our AKS cluster has nothing to do with volume snapshots failing. (We do not have connectivity issues to blob storage and use it heavily)

There is not a configuration issue with the CSI driver or VolumeSnapshotClass as they work sometimes and not others.
Markapuram Sudheer Reddy 990 Reputation points Microsoft Vendor

2025-02-25T22:03:47.93+00:00

Hi Jeff Mealo,

I just wanted to follow up regarding below response to your query.

If you have any questions, please do let us know, we will help you.

1 answer

Markapuram Sudheer Reddy 990 Reputation points Microsoft Vendor

2025-02-24T21:30:15.34+00:00
Hi Jeff Mealo,

Thank You for the response.

Please check the api server logs errors if there are issues with the CSI driver's interaction with the API server; kubectl logs -f <kube-apiserver-pod-name> -n kube-system

Try to inspect snapshot controller logs for any errors related to snapshot operations:

kubectl get pods -n kube-system | grep snapshot-controller kubectl logs -f <snapshot-controller-pod-name> -n kube-system

Please check below documentation for more information- https://github.com/kubernetes-csi/external-snapshotter/issues/346

If you have any queries, please do let us know, we will help you. If you find any logs, please share here, it will help us to investigate further.
Please sign in to rate this answer.
Jeff Mealo 0 Reputation points

2025-02-26T19:34:12.51+00:00

kubectl get pods -n kube-system | grep snapshot-controller returns nothing on my AKS cluster even when a volume snapshot is in progress

Jeff Mealo 0 Reputation points

2025-02-26T19:35:03.68+00:00

kubectl get pods -n kube-system | grep snapshot-controller doesn't return anything on my AKS cluster even when a volume snapshot is in progress

Jeff Mealo 0 Reputation points

2025-02-26T19:35:29.7166667+00:00

There is a minute delay in when you post something to when it appears. The Q&A system is very buggy.

Markapuram Sudheer Reddy 990 Reputation points Microsoft Vendor

2025-02-27T00:04:43.9033333+00:00

Hi Jeff Mealo,

We have noticed that you have rated an answer as not helpful. We appreciate your feedback and are committed to improving your experience with the Q&A.

Check CSI driver pods running in your cluster and inspect the logs of these pods. https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/csi-debug.md

Ensure snapshot controller is deployed for managing volume snapshots with CSI drivers and ensure necessary CRDs are deployed in your AKS cluster. https://github.com/kubernetes-csi/external-snapshotter

If the Diagnostic Settings is enabled for the AKS cluster, can you please share the relevant logs to better assist further. And also, please share error logs you received during troubleshooting, so that it will help us better understanding.

Jeff Mealo 0 Reputation points

2025-02-27T00:10:22.7766667+00:00

Please stop using AI. Did you test whether those commands work on an AKS 1.29 cluster? (Hint: They don't)

Arko 225 Reputation points Microsoft Vendor

2025-02-28T09:46:16.0133333+00:00

Hello Jeff Mealo, I see your issue persists and the above provided solution did not work for you. I am just consolidating it here to have a better overview of the current scenario.

Azure Disk CSI driver logs show nothing relevant, there is no issue with connectivity to Blob Storage and volume snapshots work sometimes and fail randomly, indicating an intermittent issue rather than a misconfiguration correct me if I have missed out something here?

As per my understanding, since volume snapshots sometimes works and sometimes fail, and you have already tried most of the troubleshooting, I am suspecting intermittent Azure snapshot API failures (rate limiting)

Did you get a chance to check if the AKS control plane setting is fine? that affects snapshots.

I was actually trying to try the same from my end and in that process, I created an aks cluster similar to yours and deployed a test pod on it

Now what I did from my end is I manually created a snapshot of it to check if it works manually.

If this fails, then the issue is at Azure's API level which requires product team's intervention.

Cross check this from your end once and then we can have a better clarity on the issue.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Volume Snapshots Fail on AKS with 502 and/or timeout or HTML error page

1 answer

Your answer