Network latency between Azure Global VNet Peering

SHAKIR SHAIKH 0

Hi Team, I have a scenario below. Users at the East US site access the webpage site1.abc.com which is hosted on a Citrix Netscaler in the Central US region. Users from the East US site connect in multiple ways, through VPN or AVD environment to access the webpage. Both the sites are connected through Azure Global VNet peering.

Users in the Central US location can access the site without any drops or network latency smoothly, which could be because the webpage is hosted on the same site. However, when users access the webpage from the East US site from either VPN/AVD environments they can access the webpage but the connectivity frequently and intermittently drops and restores automatically, at times the latency is too high while accessing the webpage.

Can you guide me to isolate the issue and confirm whether it is Azure Firewall/ Global VNet peering/ application side issue? In case the issue is at the East US site what can be done to fix these network issues?

Below is the flow of how users connect to the webpage.

East US users ---> VPN/AVD ---> Az. Workload VNet----> Route to Azure Firewall VNet----> Global VNet peering between Azure Firewall VNet and Central US VNet---> Citrix Netscaler (Webpage)

User's image

UJTyagi-MSFT 390 Reputation points Microsoft Employee

2024-12-16T02:08:16.3266667+00:00
Hello @SHAKIR SHAIKH

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are facing issues with intermittent drops while accessing your website site1.abc.com and traffic path is given as below.

East US users ---> VPN/AVD ---> Az. Workload VNet----> Route to Azure Firewall VNet----> Global VNet peering between Azure Firewall VNet and Central US VNet---> Citrix Netscaler (Webpage)

Since the issue is intermittent it is advised to perform the troubleshooting during the time when you experience slowness on the site.

Also, it is also important to fist identify at which layer issue is happening as you correctly mentioned in your question. Hence i would request you perform the below Linux commands from a machine in East US site.

end=$((SECONDS+300)); while [ $SECONDS -lt $end ]; do curl -w "dns_resolution: %{time_namelookup}, tcp_established: %{time_connect}, ssl_handshake_done: %{time_appconnect}, TTFB: %{time_starttransfer}\n" -o /dev/null -s https://site1.abc.com; sleep 1; done

This command will run in loop for 5 minutes and will execute after every 1 second sleep timer you may adjust the timers.

These commands will share you the time taken to dns resolution, 3 way tcp established time (Network layer) , SSL Handshake time ( TLS layer) and TTFB( Application layer ).

During slowness these timers will assist us to identify the Layer where we need to focus for our further troubleshooting.

During the issue time we can test the Global Vnet peering connectivity using the Network Watcher kindly refer the below page for help -

https://learn.microsoft.com/en-us/azure/network-watcher/connection-troubleshoot-portal

It is also advised to run the Packet captures on the Source client VM , Azure Firewall and Server where site is hosted simultaneously.

If you have Wireshark preinstalled, you may use the same or you may refer the below page to capture without installing the Wireshark.

https://techcommunity.microsoft.com/t5/iis-support-blog/capture-a-network-trace-without-installing-anything-amp-capture/ba-p/376503#:~:text=1.%20Open%20an%20elevated%20command%20prompt%20and%20run%3A,a%20temp%20directory%20or%20choose%20another%20location%29.%202.

Kindly update the outcome once you have required logs so that we can analyze further to understand the issue.
KapilAnanth-MSFT 48,081 Reputation points Microsoft Employee

2024-12-16T12:46:56.0966667+00:00
@SHAKIR SHAIKH ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

May I ask how you are isolating this to be a network issue?

Both in terms of connectivity and latency

It is possible that the application/server is over utilized or the issue is application related.

With that said,

1 . Instead of testing this with remote devices/users, I would suggest you deploy a dummyVM in the AzureEastVnet and try to access the site via VNET Peering.

If the issue prevails, we can eliminate that the the issue is related to S2S or AVD.

You can run a TCPPing or traceroute during the time of the issue and see where the latency is.

2 . If the issue is not happening in this dummyVM,

Is this a new set up? (AzureEast users)

If this was an existing set up, did you make any recent changes that resulted in the latency/connectivity issue?

Do you observer a pattern/time frame during which the issue is prevalent?

Are you seeing any application logs for the same time frame.

What is the latency you are experiencing when the issue occurs vs when the issue does not occur?

NOTE :

If you'd like to know the expected latency between regions over Microsoft backbone network (not VPN over Internet), you can refer to Azure network round-trip latency statistics

Cheers,

Kapil
SHAKIR SHAIKH 0 Reputation points

2024-12-16T21:29:56.6166667+00:00

@UJTyagi-MSFT We do not have any Linux VMs accessing the site. Only users who connect through VPN or AVD are windows hosts.
If you can share the command that can run on windows client machines, then I can provide the logs.
SHAKIR SHAIKH 0 Reputation points

2024-12-17T10:09:46.4366667+00:00
Hi KapilAnanth-MSFT,

Thanks for the getting back. I'm still unsure if this is a network latency/application related issue and hence, want to isolate it. Regarding answers to your questions PFB:

Both in terms of connectivity and latency

No issue with connectivity is observed. After accessing the application for like 40mins - 1 Hr all the application users from the EU site experience session freezing issue and eventually the session disconnects and resumes in couple of mins.

It is possible that the application/server is over utilized or the issue is application related.

Citrix netscaler which is acting as LB, has 2 backend hosts in active-passive mode and the Central US team confirms that utilization isn't even reaching 50% on the host.

1 . Instead of testing this with remote devices/users, I would suggest you deploy a dummyVM in the AzureEastVnet and try to access the site via VNET Peering.

If the issue prevails, we can eliminate that the the issue is related to S2S or AVD.

I'm going to deploy another testing VNET and test the connectivity and will share the results with you.

You can run a TCPPing or traceroute during the time of the issue and see where the latency is.

TCPPing and traceroute isn't helping much as the latency/connectivity drop is experienced only when accessing the application.

Is this a new set up? (AzureEast users)

The setup has been in place for a couple of months but issue started to arise when the number of users who are accessing the application started to increase.

If this was an existing set up, did you make any recent changes that resulted in the latency/connectivity issue?

No changes were made, only new users were onboarded to access the application.

Do you observer a pattern/time frame during which the issue is prevalent?

During peak hours, when most of the users are accessing the application. No issues observed when no one is accessing the application.

Are you seeing any application logs for the same time frame.

Working to see if application logs can be accessible from citrix.

What is the latency you are experiencing when the issue occurs vs when the issue does not occur?

While accessing application - 4-7 secs latency is observed.

When issue is not observed - 20-30 ms.
KapilAnanth-MSFT 48,081 Reputation points Microsoft Employee

2024-12-17T11:43:52.55+00:00
@SHAKIR SHAIKH ,

Sure, let us know how this goes.

Wrt, "TCPPing and traceroute isn't helping much as the latency/connectivity drop is experienced only when accessing the application."

Does this mean, during the time of the issue when you perform TCPPing and traceroute to the server, you are not facing any latency?

Only access to the Application is slow?

In this case, I believe this is totally isolated to L7 (Of the application).

The next steps here would be to collect packet captures

Though they are encrypted, I believe we can use the TCP Delta to see if there is a network issue

If you were to collect captures, make sure you do this at every hop feasible.

Cheers,

Kapil

Share via

Network latency between Azure Global VNet Peering

Your answer