Understand the routes propagation in Azure

Question

I'm investigating route propagation behavior in an Azure Hub-Spoke environment with an ExpressRoute gateway. My Hub VNet connects to on-premises networks via ExpressRoute. I have a VM in a spoke VNet.

I've noticed discrepancies between the routes learned by the VNet gateway (obtained using Get-AzVirtualNetworkGatewayLearnedRoute) and the effective routes on the VM's NIC. Specifically, the Next Hop for a specific destination VNet (let's call it 'destvm') differs.

The VNet gateway learns the route to 'destvm' with the Next Hop as the gateway subnet (as expected). However, the effective routes on the VM's NIC show a different Next Hop IP address, which I cannot identify within Azure.

Can you explain how route propagation works in this scenario, considering factors like Route Table entries, VNet peering configuration, and potential influences from Network Security Groups? How can I effectively troubleshoot and debug these route discrepancies?

Pasting snips below for clarity:

1. Discrepency in next hop IP

A basic diagram to understand the traffic flow

Thanks a lot in advance!

Answer 1

Hello Apurva Pathak
Greetings!

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you need clarification on how route propagation works check the blow information:

Route Propagation in Azure

1. Route Propagation Basics:
   • In Azure, when you have a Hub VNet connected to on-premises networks via ExpressRoute, the VNet gateway learns routes from the on-premises network and propagates them to the connected VNets (spokes).
     • The routes learned by the VNet gateway can include routes to specific address spaces in the spoke VNets, which are then used to determine how traffic is routed.
2. Effective Routes on the VM's NIC:
   • The effective routes on a VM's NIC are determined by several factors:
   • User -Defined Routes (UDRs): If you have defined any UDRs in the Hub or Spoke VNets, they can override the default routing behaviour.
   • Route Propagation: Routes learned from the VNet gateway (including those from ExpressRoute) are propagated to the spoke VNets.
   • VNet Peering: If the spoke VNets are peered with the Hub VNet, the routes from the Hub can be propagated to the spokes.
   • Network Security Groups (NSGs): While NSGs do not directly affect routing, they can impact connectivity by allowing or denying traffic based on security rules.

Clarifying the Discrepancy

Next Hop IP Address:

The Next Hop IP address for a specific destination (like 'destvm') may differ between the learned routes and the effective routes on the VM's NIC due to the following reasons:

• Route Table Entries: If there are UDRs in place that direct traffic to a specific next hop (e.g., a Network Virtual Appliance or another gateway), this can change the effective route.
• VNet Peering Configuration: If the spoke VNet is peered with another VNet that has a different routing configuration, this can also affect the effective routes.
• Dynamic Routing: The Azure routing infrastructure may dynamically adjust routes based on the current state of the network, which can lead to differences in the next hop.

Identifying the Next Hop:

1. If the Next Hop IP address on the VM's NIC is not identifiable within Azure, it may be due to:
   • A misconfiguration in the routing setup.
   • A route that is being learned from a different source (e.g., another VNet or a VPN connection).
   • A potential issue with route propagation or the way routes are being advertised.

Note:

1. The Express Route or Express Route Gateway operates on BGP, automatically learning routes in Azure and advertising them to the on-premises network. When checking the NIC effective route of the destination VM, you will see the on-premises route connected via Express Route, displaying the virtual network gateway as the next hop and the IP as the BGP peered IP of Express Route. If there is no route table for the destination VM subnet, it will only show the BGP peer IP in the destination VM NIC effective routes.
2. If the next hop is displayed as an IP, you need to verify the UDR or route table. Similarly, the peering option between HUB and spoke VNETS must be correctly verified.
3. Azure will always prefer the shortest address prefix in UDRs.

Troubleshooting Steps

1. Check Route Tables:
   • Use the Azure portal or PowerShell to review the route tables associated with the Hub and Spoke VNets. Look for any UDRs that might be affecting the routing behavior.
2. Review VNet Peering Configuration:
   • Ensure that the VNet peering settings are correctly configured to allow for route propagation. Check if "Allow forwarded traffic" and "Allow gateway transit" are enabled as needed.
3. Use Azure Network Watcher:
   • Utilize Azure Network Watcher to diagnose network issues. The Connection Troubleshoot feature can help you understand how traffic is routed and whether it is being blocked.
4. Examine NSGs:
   • Review the NSGs associated with the VM and the subnets to ensure that there are no rules that might be blocking traffic to the destination.
5. Check Learned Routes:
   • Use the Get-AzVirtualNetworkGatewayLearnedRoute command to review the routes learned by the VNet gateway. Compare these with the effective routes on the VM's NIC to identify any discrepancies.
6. Investigate the Next Hop IP:
   • If the Next Hop IP is not recognized, try to trace the route using tools like tracert or ping to see where the traffic is being directed. This can help identify if the traffic is being routed through an unexpected path.

By following these troubleshooting steps, you should be able to identify the cause of the discrepancies in routing behavior between the learned routes and the effective routes on the VM's NIC. Understanding the interplay between UDRs, route propagation, VNet peering, and NSGs is crucial in diagnosing routing issues in Azure. If you continue to experience difficulties, consider reaching out to Azure support for more in-depth analysis and assistance.

• When using an ExpressRoute circuit, BGP is enabled by default. Through BGP, only on-premises and Azure routes will be advertised and learned.
• To verify this, use the connection troubleshoot option in the Virtual Machine under the help section in the Azure portal, which will display all hops from source to destination.
• When checking the NIC effective route of the destination VM, you will see the on-premises route connected via ExpressRoute, displaying the virtual network gateway as the next hop and the IP as the BGP peered IP of ExpressRoute. If there is no route table for the destination VM subnet, it will only show the BGP peer IP in the destination VM NIC effective routes. If the next hop is displayed as an IP, you need to verify the UDR or route table. Similarly, the peering option between HUB and spoke VNETs must be correctly verified. Azure will always prefer the shortest address prefix in UDRs.
• BGP Route Propagation should be set to “Enabled” on the Gateway Subnet to ensure the availability of the gateway. If this is set to disabled, the gateway will not function.
• Ensure that the next hop IP addresses are correctly set up in the BGP configuration of the target NVAs. This is crucial for maintaining proper routing and avoiding discrepancies.

Kindly check the below reference documents for more understanding:

Reference doc how BGP will work: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview#border-gateway-protocol
Routing doc: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview
Vent peering doc: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview#gateways-and-on-premises-connectivity

---

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please remember to "Accept Answer" if answer helped, so that others in the community facing similar issues can easily find the solution.

Best Regards,

Praveen