Issue: High nat gateway data transfer
We are running ephemeral workloads that creates 2000-3000 pods and corresponding to that autoscaler spins up around 500 odd nodes. Now even though we are pulling our application images from ACR via service endpoints , we still see a 60-70gb data transfer via nat gateway.
Our dynamic workloads run continuously , creates 500 nodes , nodes go down and come up again.
This has spiked up our nat gateway costs.
We suspect this is due to AKS data nodes pulling system pod images from microsoft container registry mcr.microsoft.com . Is there a way to pull these privately ?
Or do we have a way where we can store these images on ACR and modify kube-system daemonsets to pull images from our hosted ACR privately?
This is impacting all our customers. Please help.
Below are the microsoft images that i see on each node:
- mcr.microsoft.com/aks/aks-node-ca-watcher
mcr.microsoft.com/aks/ip-masq-agent-v2
mcr.microsoft.com/aks/msi/addon-token-adapter
mcr.microsoft.com/azure-policy/policy-kubernetes-addon-prod
mcr.microsoft.com/azure-policy/policy-kubernetes-webhook
mcr.microsoft.com/azuremonitor/containerinsights/ciprod/prometheus-collector/images
mcr.microsoft.com/azuremonitor/containerinsights/ciprod
mcr.microsoft.com/cbl-mariner/busybox
mcr.microsoft.com/containernetworking/azure-cni
mcr.microsoft.com/containernetworking/azure-cns
mcr.microsoft.com/containernetworking/azure-ipam
mcr.microsoft.com/containernetworking/azure-npm
mcr.microsoft.com/containernetworking/cni-dropgz
mcr.microsoft.com/mirror/docker/library/busybox
mcr.microsoft.com/oss/azure/secrets-store/provider-azure
mcr.microsoft.com/oss/calico/cni
mcr.microsoft.com/oss/calico/node
mcr.microsoft.com/oss/calico/pod2daemon-flexvol
mcr.microsoft.com/oss/cilium/cilium
mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi
mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi
mcr.microsoft.com/oss/kubernetes-csi/blob-csi
mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar
mcr.microsoft.com/oss/kubernetes-csi/livenessprobe
mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver
mcr.microsoft.com/oss/kubernetes/apiserver-network-proxy/agent
mcr.microsoft.com/oss/kubernetes/autoscaler/addon-resizer
mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler
mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager
mcr.microsoft.com/oss/kubernetes/coredns
mcr.microsoft.com/oss/kubernetes/kube-proxy
mcr.microsoft.com/oss/kubernetes/kube-state-metrics
mcr.microsoft.com/oss/kubernetes/metrics-server
mcr.microsoft.com/oss/kubernetes/pause
mcr.microsoft.com/oss/kubernetes/windows-gmsa-webhook
mcr.microsoft.com/oss/nvidia/k8s-device-plugin
mcr.microsoft.com/oss/open-policy-agent/gatekeeper