Why does Azure virtual machine with dual stack network is not able to reach IMDS server 169.254.169.254

Mahudeeswaran Palanisamy 0

We have our appliance image which is basically a linux machine deployed using Azure SDK (Java), the virtual machine is configured with two network interfaces, both of which is connected to dual stack subnet. After the node is deployed and booting up, as part of cloud-init, it connects to IMDS server to get the metadata, but the request is timing out, logs below.

[ 13.248532] cloud-init[493]: /usr/lib/python3/dist-packages/requests/init.py:87: RequestsDependencyWarning: urllib3 (2.2.3) or chardet (4.0.0) doesn't match a supported version!

[ 13.255596] cloud-init[493]: warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

[ OK ] Listening on Load/Save RF …itch Status /dev/rfkill Watch.

[ 13.464770] cloud-init[619]: Cloud-init v. 23.1.2-0ubuntu0~22.04.1 running 'init-local' at Mon, 16 Dec 2024 06:05:28 +0000. Up 13.44 seconds.

***[ 45.738883] cloud-init[619]: 2024-12-16 06:06:00,320 - azure.py[WARNING]: Failed to fetch metadata from IMDS: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/instance?api-version=2021-08-01&extended=true (Caused by ConnectTimeoutError(

Sai Krishna Katakam 1,710 Reputation points Microsoft Vendor

2025-01-08T07:29:49.0033333+00:00
Hi Mahudeeswaran Palanisamy,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

Azure virtual machines with a dual stack network (IPv4 and IPv6) may face connectivity issues with the Instance Metadata Service (IMDS) at 169.254.169.254 due to routing conflicts or misconfigurations. The IMDS is accessible only via IPv4, and if the network configuration prioritizes IPv6, it may prevent successful communication with the IMDS.

Troubleshooting Steps:

Ensure that the network interface is configured to allow IPv4 traffic.

Verify that there are no routing conflicts that could affect access to the IMDS.

Use the following command to test connectivity to the IMDS;

curl -H "Metadata: true" "http://169.254.169.254/metadata/instance?api-version=2021-08-01&extended=true"

A successful response (HTTP status code 200) indicates that the IMDS is reachable.

Ensure that there are no firewall rules blocking outbound traffic to the IMDS IP address (169.254.169.254).

If IPv6 is not required for your application, consider disabling it on the network interface to avoid potential conflicts.

For more information, please refer to the below documentation:
Azure Instance Metadata Service

If an answer has been helpful, please consider accept the "Answer" and "Upvote" to help increase visibility of this question for other members of the Microsoft Q&A community.
Mahudeeswaran Palanisamy 0 Reputation points

2025-01-08T11:28:04.34+00:00

Hi @Sai Krishna Katakam ,

As I have mentioned in my earlier comment, I tried the CURL already and in that, the request was going out of the VM, so it means there shouldn't be routing conflict causing the issue I think.

I also mentioned about the network configuration, the network has IPv4 and IPv6 address, and the request was sent from the primary IP address.

Again as I mentioned earlier, another VM in same virtual network is working, no firewall is configured.

IPv6 is required for the application.

If the same image is deployed via Azure portal, it is working. Issue is observed only when we try deploying via Azure Java SDK and the VM has dual stack network.

PS: My earlier comment is missing now, in which I have added details about what I have tried and the difference observed between working and non working setup, with screen shots. Could you please help to restore it. And also observing some part of my question is also edited/missing.
Sai Krishna Katakam 1,710 Reputation points Microsoft Vendor

2025-01-09T04:02:20.65+00:00
Hi Mahudeeswaran Palanisamy,

We noticed your feedback that the above answer was not helpful. Thank you for taking time to share feedback.

Based on your clarification, it appears that the issue is specific to deployments made using the Azure Java SDK with dual-stack networks. Here are some further insights and troubleshooting steps:

Verify the route to 169.254.169.254 using:

ip route get 169.254.169.254

Make sure it points to the primary NIC's IPv4 address.

If the route is missing or incorrect, add it manually:

sudo ip route add 169.254.169.254 dev <primary-nic-name>

Confirm the NIC's MTU is correctly set (typically 1500 for Azure networks):

ip link show <primary-nic-name>

Add retry logic or a delay in cloud-init to ensure the network is fully initialized before accessing IMDS.

Compare the VM and network configuration between SDK and portal-deployed VMs to identify discrepancies in NIC bindings, routes, or MTU settings.

Ensure you're using the latest Azure Java SDK, as earlier versions may have known issues with dual-stack provisioning.If you have any further queries, do let us know. If the comment is helpful, please click "Accept Answer" and "Upvote".
Mahudeeswaran Palanisamy 0 Reputation points

2025-01-09T08:52:52.96+00:00
Hi @Sai Krishna Katakam ,

Many thanks for your response.

Verify the route to 169.254.169.254 using:

Verified, it points to the primary nic's ipv4 address.

gigamon@gigavue-vseries-node:~$ ip route get 169.254.169.254169.254.169.254 via 10.0.1.1 dev eth0 src 10.0.1.8 uid 1000

If the route is missing or incorrect, add it manually:

Route is already there. Tried tcpdump, TCP SYN acket is sent from the node, but no response is received.

Confirm the NIC's MTU is correctly set (typically 1500 for Azure networks):

gigamon@gigavue-vseries-node:~$ ip link show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 60:45:bd:00:a9:5f brd ff:ff:ff:ff:ff:ff gigamon@gigavue-vseries-node:~$ ip route get 169.254.169.254169.254.169.254 via 10.0.1.1 dev eth0 src 10.0.1.8 uid 1000

MTU is set properly, 1500.

Add retry logic or a delay in cloud-init to ensure the network is fully initialized before accessing IMDS.

It was retrying for around 30 mins, but still the issues observed.

Compare the VM and network configuration between SDK and portal-deployed VMs to identify discrepancies in NIC bindings, routes, or MTU settings.

Checked the network configuration, as mentioned earlier, only difference I could see is, in working case, though the network is dual stack there is only IPv4 address assigned to the interface, no IPv6 address is assigned, just it only has IPv6 link local address. Where as in non-working case, interface has proper IPv4, IPv6 address assigned and it also had IPv6 link local address.

Another thing is the route, non-working case had IPv6 route, but working case didn't had initially. Tried adding IPv6 manually, still it is working with IPv6 and it's route entry in working node.

Are you looking for any specific parameter to be checked, if so please let me know the params to be checked.

Ensure you're using the latest Azure Java SDK, as earlier versions may have known issues with dual-stack provisioning.

Will try it, but one thing to note is, our earlier released images, which was working until recently is also facing this issue and stopped working, because of which I didn't try that. Did we had any such issue in earlier releases, if so it would be helpful if you could point to it.

I'm trying to deploy a Ubuntu server via SDK from our code to check if the issue observed in that as well. Please let me know if you have any other suggestions to try.
Sai Krishna Katakam 1,710 Reputation points Microsoft Vendor

2025-01-10T04:40:36.97+00:00

Hi Mahudeeswaran Palanisamy,

Thanks for your reply. I'm working on this issue and will get back to you as soon as possible.

In the meantime, please refer to the following link, which might be helpful: Access Azure Instance Metadata Service (169.254.169.254) from Windows Docker Container

Thank you.
Sai Krishna Katakam 1,710 Reputation points Microsoft Vendor

2025-01-13T09:11:47.18+00:00

Hi Mahudeeswaran Palanisamy,

Could you please confirm whether your issue has been resolved? If not, please provide the latest errors or logs related to the issue you're facing so we can assist you further.

Thank you.
Sai Krishna Katakam 1,710 Reputation points Microsoft Vendor

2025-01-20T10:18:09.6233333+00:00

Hi Mahudeeswaran Palanisamy,

Just following up to check if your issue has been resolved. If not, please share any updates, recent logs, or errors you're encountering so we can continue assisting you. Let us know how we can help further!

Thank you.
Mahudeeswaran Palanisamy 0 Reputation points

2025-01-28T09:53:14.74+00:00

Hi @Sai Krishna Katakam ,

This issue is not resolved yet. Have raised a Azure support case for this. Working with them.

Have tried deploying an Ubunutu VM, faced the same issue. Tried deploying a VM using latest Java SDK version today, facing the issue.Thanks,

Mahudees
Mahudeeswaran Palanisamy 0 Reputation points

2025-01-28T11:04:43.27+00:00

Please refer previous comment.
Nikhil Duserla 4,280 Reputation points Microsoft Vendor

2025-01-30T12:00:35.3766667+00:00

Hi @Mahudeeswaran Palanisamy,Thank you for letting us know.

1 answer

Silvia Wibowo 5,281 Reputation points Microsoft Employee

2025-01-07T21:58:23.95+00:00

Hi @Mahudeeswaran Palanisamy , I understand you have an appliance with dual NIC (network interface card) deployed as Azure VM and it couldn't reach IMDS (Instance Metadata Service).

Please note: IMDS requests must be sent using VM's primary NIC and primary IP, and DHCP must be enabled.

Reference: Azure Instance Metadata Service

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Why does Azure virtual machine with dual stack network is not able to reach IMDS server 169.254.169.254

1 answer

Your answer