How do I allow pods to access the IMDS when using IPVS as kube-proxy scheduler?

Question

How do I allow pods to access the IMDS when using IPVS as kube-proxy scheduler?

Orestis Gklavas 0

We are using an AKS cluster, whose services still use AAD Pod Identity (yes, I know it is deprecated, and yes, it will be migrated to Workload Identity soon).

Due to some performance issues, I decided to try setting kube-proxy to use the IPVS scheduler instead of the default iptables. Indeed, the performance was much higher. However, as I became painfully aware when deploying a nodepool, the NMI pods were unable to contact IMDS to get the required tokens, so my pods would hang at the init state.

I attempted to toy around with kube-proxy IPVS paramters such as excludeCIDRs, with 169.254.169.254/32 as the targt CIDR, but these don't seem to get added to the daemonset's command line params (no configmap is present in the kube-system namespace).

In addition, the "configure kube-proxy" page ( https://learn.microsoft.com/en-us/azure/aks/configure-kube-proxy) points to the AKS Cluster Schema which is supposed to contain the full kube-proxy configuration structure, but I can't seem to be able to find this structure.

Any ideas are welcome. Thanks for taking the time to read through this.

Pramidha Yathipathi 10 Reputation points Microsoft External Staff

2025-03-11T07:47:31.21+00:00
Hi Orestis Gklavas,

Switching to IPVS improves performance, but it also changes how traffic is routed, which can cause issues with IMDS access. Since AAD Pod Identity relies on NMI pods reaching IMDS (169.254.169.254), you’ll need to make sure this traffic isn’t affected by IPVS rules.

1.Exclude IMDS from kube-proxy rules

Since AKS doesn’t use a ConfigMap for kube-proxy, you’ll need to set excludeCIDRs during cluster updates. You can try:

az aks update \ --resource-group <your-resource-group> \ --name <your-cluster-name> \ --kube-proxy-config '{"mode":"ipvs","excludeCIDRs":["169.254.169.254/32"]}'

However, this might not always work as expected in AKS.

Manually adjust iptables on nodes If the above doesn’t work, you can manually add an iptables rule to ensure traffic to IMDS is handled properly:

iptables -t nat -A PREROUTING -d 169.254.169.254/32 -j RETURN

This should allow traffic to bypass IPVS.

3.Run NMI pods with hostNetwork: true

Modifying the NMI deployment to use hostNetwork: true can help, as it allows the pods to bypass kube-proxy and communicate directly with IMDS.

Please refer the below documents:

https://learn.microsoft.com/en-us/azure/virtual-machines/instance-metadata-service?tabs=windows

https://learn.microsoft.com/en-us/azure/aks/imds-restriction

If the comment was helpful, please don't forget to click "Upvote".

If you have any further queries, please let us know we are glad to help you.

Thank You.
Orestis Gklavas 0 Reputation points

2025-03-11T12:48:03.14+00:00

Hi @Anonymous ,

Well, it seems solution 2 is a bit tricky, since - at least when using the debug pod that Microsoft suggests for connecting to the node - I have no permission to modify iptables. Perhaps I'll give it a shot with direct ssh, however I am not sure about the persistence.

As for the hostNetwork: true in the NMI pods, it was already set.

In any case, your first solution pointed me to something: you have excludedCIDRs as a top level setting, while I was attempting to do the same thing by adding it inside the ipvsConfig block. In either case, this setting does not appear when I describe or get the kube-proxy daemonSet, so I can't be sure if the setting is somehow applied, however it seems to be able to communicate now. If you have any insights on how I can verify if exludedCIDRs has been applied, I'd be grateful.

UPDATE: Not sure why it seemed to work, however when I applied the config to the main cluster the pods still cannot access IMDS. So I guess excludedCIDRs is unsupported.
Pramidha Yathipathi 10 Reputation points Microsoft External Staff

2025-03-12T10:50:17.0533333+00:00
Hi Orestis Gklavas,

Seems like excludedCIDRs isn’t working as expected in AKS. Since it’s not showing up in the kube-proxy DaemonSet,try checking the logs instead:

kubectl logs -n kube-system -l k8s-app=kube-proxy

If excludedCIDRs was applied, you should see some mention of it in there.

– If excludedCIDRs really isn’t working, one option could be setting hostNetwork: true on the application pods that need IMDS access (not just the NMI pods).

– If you get SSH access and can modify iptables.

If the comment was helpful, please don't forget to click "Upvote".

If you have any further queries, please let us know we are glad to help you.

Thank You.

Your answer

Pramidha Yathipathi 10 Reputation points Microsoft External Staff

2025-03-11T07:47:31.21+00:00

Hi Orestis Gklavas,

Switching to IPVS improves performance, but it also changes how traffic is routed, which can cause issues with IMDS access. Since AAD Pod Identity relies on NMI pods reaching IMDS (169.254.169.254), you’ll need to make sure this traffic isn’t affected by IPVS rules.

1.Exclude IMDS from kube-proxy rules

Since AKS doesn’t use a ConfigMap for kube-proxy, you’ll need to set excludeCIDRs during cluster updates. You can try:

az aks update \ --resource-group <your-resource-group> \ --name <your-cluster-name> \ --kube-proxy-config '{"mode":"ipvs","excludeCIDRs":["169.254.169.254/32"]}'

However, this might not always work as expected in AKS.

Manually adjust iptables on nodes If the above doesn’t work, you can manually add an iptables rule to ensure traffic to IMDS is handled properly:

iptables -t nat -A PREROUTING -d 169.254.169.254/32 -j RETURN

This should allow traffic to bypass IPVS.

3.Run NMI pods with hostNetwork: true

Modifying the NMI deployment to use hostNetwork: true can help, as it allows the pods to bypass kube-proxy and communicate directly with IMDS.

Please refer the below documents:

https://learn.microsoft.com/en-us/azure/virtual-machines/instance-metadata-service?tabs=windows

https://learn.microsoft.com/en-us/azure/aks/imds-restriction

If the comment was helpful, please don't forget to click "Upvote".

If you have any further queries, please let us know we are glad to help you.

Thank You.
Orestis Gklavas 0 Reputation points

2025-03-11T12:48:03.14+00:00

Hi @Anonymous ,

Well, it seems solution 2 is a bit tricky, since - at least when using the debug pod that Microsoft suggests for connecting to the node - I have no permission to modify iptables. Perhaps I'll give it a shot with direct ssh, however I am not sure about the persistence.

As for the hostNetwork: true in the NMI pods, it was already set.

In any case, your first solution pointed me to something: you have excludedCIDRs as a top level setting, while I was attempting to do the same thing by adding it inside the ipvsConfig block. In either case, this setting does not appear when I describe or get the kube-proxy daemonSet, so I can't be sure if the setting is somehow applied, however it seems to be able to communicate now. If you have any insights on how I can verify if exludedCIDRs has been applied, I'd be grateful.

UPDATE: Not sure why it seemed to work, however when I applied the config to the main cluster the pods still cannot access IMDS. So I guess excludedCIDRs is unsupported.
Pramidha Yathipathi 10 Reputation points Microsoft External Staff

2025-03-12T10:50:17.0533333+00:00

Hi Orestis Gklavas,

Seems like excludedCIDRs isn’t working as expected in AKS. Since it’s not showing up in the kube-proxy DaemonSet,try checking the logs instead:

kubectl logs -n kube-system -l k8s-app=kube-proxy

If excludedCIDRs was applied, you should see some mention of it in there.

– If excludedCIDRs really isn’t working, one option could be setting hostNetwork: true on the application pods that need IMDS access (not just the NMI pods).

– If you get SSH access and can modify iptables.

If the comment was helpful, please don't forget to click "Upvote".

If you have any further queries, please let us know we are glad to help you.

Thank You.

Share via

How do I allow pods to access the IMDS when using IPVS as kube-proxy scheduler?

Your answer