Connectivity issue with API server

Question

Hi Team,

We are facing an issue with connectivity to our AKS API Server. Certain services that we have on our cluster - akv2k8s and cert-manager. Please note that we have a feature to start/stop our clusters and this issue surfaced today morning once the cluster started up. We have been doing this activity since long without any issues in the past.

Below are some of the logs that we see on our pods -

akv2k8s controller-

Trace[1452107515]: ---"Objects listed" error:Get "https://10.2.0.1:443/api/v1/configmaps?limit=500&resourceVersion=0": EOF 10049ms (14:00:17.752)

Trace[1452107515]: [10.049223236s] [10.049223236s] END

E0227 14:00:17.752183 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://10.2.0.1:443/api/v1/configmaps?limit=500&resourceVersion=0": EOF

W0227 14:00:38.915961 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Secret: Get "https://10.2.0.1:443/api/v1/secrets?limit=500&resourceVersion=0": EOF

I0227 14:00:38.916016 1 trace.go:236] Trace[784576806]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229 (27-Feb-2025 14:00:28.865) (total time: 10050ms):

Trace[784576806]: ---"Objects listed" error:Get "https://10.2.0.1:443/api/v1/secrets?limit=500&resourceVersion=0": EOF 10050ms (14:00:38.915)

Trace[784576806]: [10.050356026s] [10.050356026s] END

akv2k8s injector -

0227 14:02:04.144847 1 pod.go:295] "creating client certificate to use with auth service" dev/="(MISSING)"

I0227 14:02:04.201289 1 pod.go:303] "create authentication service secret" dev/="(MISSING)"

E0227 14:02:04.203713 1 main.go:162] "failed to mutate" err="Post "https://10.2.0.1:443/api/v1/namespaces/dev/secrets": EOF" pod="dev/"

2025/02/27 14:02:04 [ERROR] admission webhook error: Post "https://10.2.0.1:443/api/v1/namespaces/dev/secrets": EOF

cert-manager-

E0227 14:02:02.048219 1 leaderelection.go:436] error retrieving resource lock kube-system/cert-manager-controller: Get "https://10.2.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cert-manager-controller": EOF

Kindly help us with finding the issue and resolving it.

Answer 1

Hi Vipal Mehta,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

Based on logs and context you provided, the issue seems to be connectivity to the Kubernetes API server (10.2.0.1:443) after cluster was started. The logs show EOF errors when trying to reach (10.2.0.1:443) which indicates the API server is not responding.

You can find below troubleshooting steps to check the connectivity to the AKS cluster's API server:

With the FQDN, check whether the API server is reachable from the client machine by using the name server lookup (nslookup), client URL (curl), and telnet commands:

Check if the DNS Resolution is working: $ nslookup <cluster-fqdn>

Then check if the API Server is reachable:

$ curl -Iv https://<cluster-fqdn>
$ telnet <cluster-fqdn> 443

Misconfigurations in the API server components or flags can also lead to connectivity issues. Check the kube-apiserver logs for any errors or warnings:

journalctl -u kube-apiserver -l

Resource constraints on the master node can impair the API server's functionality. Verify that the master node has sufficient resources (CPU, memory, disk) available:

kubectl top nodes

Ensure there are no issues with service account credentials or RBAC policies that might be preventing access to the API server.

SSL/TLS certificate issues can also prevent secure communication with the API server. Ensure that the certificates are properly configured and valid. Check the expiration date and renewal status of the certificates:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate

Enter the following az aks show command in Azure CLI. This command gets the fully qualified domain name (FQDN) of your AKS cluster configuration:

az aks show --resource-group <cluster-resource-group> --name <cluster-name> --query fqdn

Please check the below documents for reference:

Certificate rotation in Azure Kubernetes Service (AKS)

Root cause and solutions

If you have any further queries, please do let us know. If the answer is helpful, please click "Accept Answer" and "Upvote it"