Connectivity issue with API server

Question

Hi Team,

We are facing an issue with connectivity to our AKS API Server. Certain services that we have on our cluster - akv2k8s and cert-manager. Please note that we have a feature to start/stop our clusters and this issue surfaced today morning once the cluster started up. We have been doing this activity since long without any issues in the past.

Below are some of the logs that we see on our pods -

akv2k8s controller-

Trace[1452107515]: ---"Objects listed" error:Get "https://10.2.0.1:443/api/v1/configmaps?limit=500&resourceVersion=0": EOF 10049ms (14:00:17.752)

Trace[1452107515]: [10.049223236s] [10.049223236s] END

E0227 14:00:17.752183 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://10.2.0.1:443/api/v1/configmaps?limit=500&resourceVersion=0": EOF

W0227 14:00:38.915961 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1.Secret: Get "https://10.2.0.1:443/api/v1/secrets?limit=500&resourceVersion=0": EOF

I0227 14:00:38.916016 1 trace.go:236] Trace[784576806]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229 (27-Feb-2025 14:00:28.865) (total time: 10050ms):

Trace[784576806]: ---"Objects listed" error:Get "https://10.2.0.1:443/api/v1/secrets?limit=500&resourceVersion=0": EOF 10050ms (14:00:38.915)

Trace[784576806]: [10.050356026s] [10.050356026s] END

akv2k8s injector -

0227 14:02:04.144847 1 pod.go:295] "creating client certificate to use with auth service" dev/="(MISSING)"

I0227 14:02:04.201289 1 pod.go:303] "create authentication service secret" dev/="(MISSING)"

E0227 14:02:04.203713 1 main.go:162] "failed to mutate" err="Post "https://10.2.0.1:443/api/v1/namespaces/dev/secrets": EOF" pod="dev/"

2025/02/27 14:02:04 [ERROR] admission webhook error: Post "https://10.2.0.1:443/api/v1/namespaces/dev/secrets": EOF

cert-manager-

E0227 14:02:02.048219 1 leaderelection.go:436] error retrieving resource lock kube-system/cert-manager-controller: Get "https://10.2.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cert-manager-controller": EOF

Kindly help us with finding the issue and resolving it.