I'm searching for possibility to implement a topology like this: Spoke VNETs <====> NVA, for instance FortiGate <=====> VPN Gateway <=====> on-premises To implement this with static routing is pretty straightforward, but has irritating limitations : If I place the NVA and the VPN Gateway in the same HUB VNET, while enabling the Enable Spoke-VNET to use HUB-VNET's remote gateway or route server option - then narrow on-premises prefixes learned from VPN Gateway will override more generic prefixes in Spoke VNET UDRs, which steer traffic towards on-premises to the NVA for inspection. I'll need to override each one of them in the UDRs, which is not practical. If I place them in the same HUB VNET without enabling this option, or if I place the VPN Gateway in a VNET not peered directly with Spokes - then the VPN Gateway won't know IP ranges of Spoke VNETs. It still will deliver traffic for them to the NVA by means of a UDR applied to its GatewaySubnet , but if the VPN Gateway will learn some prefix overlapping with Spoke VNET's prefix from one of its BGP-over-IPsec peers - it will send the traffic there instead of delivering it to the NVA, and communication will fail. Example - UDR of GatewaySubnet says " deliver traffic to 10.0.0.0.0/0 via NVA ". The Spoke VNET's range is 10.11.12.0/24 . If the VPN Gateway doesn't learn it automatically from the SDN, while learning a prefix 10.11.0.0/16 from some its BGP peer - then traffic to 10.11.12.0/24 will be sent to this peer instead of the Spoke. I'm looking for solution that would allow: Spoke VNETs to deliver traffic to the NVA - without me having to override by UDRs each narrow prefix injected to them by the VPN Gateway. VPN Gateways to learn automatically prefixes of Spoke VNETs - while still delivering traffic to them via the NVA. Is such this possible? With Route Server or without? Thanks! Mucius.

Figured out a partial solution: The idea is this: Since the best way to inject routes into VPN Gateway's internal routing table is by BGP, nothing prevents us from establishing BGP-over-IPsec peerings between the Gateway and our firewall VMs. So the firewall VMs will dynamically learn Spoke VNETs' prefixes from Azure Route Server and re-advertise them to the VPN Gateway instances. The VPN Gateway will be deployed in a separate VNET, to avoid its GatewaySubnet learning direct routes to Spokes. The main problem of this solution is that it requires Active-Passive firewall setup, to keep traffic symmetry. Documented more details here: https://github.com/cat-mucius/kb/wiki/Dynamic-routing-with-Azure-VPN-Gateway-and-with-traffic-inspection-by-custom-firewall Update: judging by the " Direct routing to NVA instances for dual-role connectivity and firewall NVAs " section in Virtual WAN documentation, Microsoft themselves didn't find a good active-active solution for this problem. :-) Azure's software-defined networking platform doesn't guarantee flow-level symmetry, meaning the inbound flow to Azure and outbound flow from Azure can land on different instances of the NVA. This results in asymmetric routing which is dropped by stateful firewall inspection. Therefore, it isn't recommended to use active-active connectivity patterns where an NVA is behaving as a dual-role NVA unless the NVA can support asymmetric forwarding or support session sharing/synchronization.

Dynamic routing with VPN Gateway and with inspection by NVA

Cat Mucius 71

I'm searching for possibility to implement a topology like this:

Spoke VNETs <====> NVA, for instance FortiGate <=====> VPN Gateway <=====> on-premises

To implement this with static routing is pretty straightforward, but has irritating limitations:

If I place the NVA and the VPN Gateway in the same HUB VNET, while enabling the Enable Spoke-VNET to use HUB-VNET's remote gateway or route server option - then narrow on-premises prefixes learned from VPN Gateway will override more generic prefixes in Spoke VNET UDRs, which steer traffic towards on-premises to the NVA for inspection. I'll need to override each one of them in the UDRs, which is not practical.
If I place them in the same HUB VNET without enabling this option, or if I place the VPN Gateway in a VNET not peered directly with Spokes - then the VPN Gateway won't know IP ranges of Spoke VNETs. It still will deliver traffic for them to the NVA by means of a UDR applied to its GatewaySubnet, but if the VPN Gateway will learn some prefix overlapping with Spoke VNET's prefix from one of its BGP-over-IPsec peers - it will send the traffic there instead of delivering it to the NVA, and communication will fail.

Example - UDR of GatewaySubnet says "deliver traffic to 10.0.0.0.0/0 via NVA".
The Spoke VNET's range is 10.11.12.0/24.
If the VPN Gateway doesn't learn it automatically from the SDN, while learning a prefix 10.11.0.0/16 from some its BGP peer - then traffic to 10.11.12.0/24 will be sent to this peer instead of the Spoke.

I'm looking for solution that would allow:

Spoke VNETs to deliver traffic to the NVA - without me having to override by UDRs each narrow prefix injected to them by the VPN Gateway.
VPN Gateways to learn automatically prefixes of Spoke VNETs - while still delivering traffic to them via the NVA.

Is such this possible? With Route Server or without?

Thanks!
Mucius.

Luis Arias 7,531 Reputation points

2024-10-18T22:13:06.6233333+00:00
Hi Cat Mucius,

I understand that you want to allow Spoke VNETs to send traffic to NVA without manual UDR overrides for each prefix from VPN Gateway and also enable VPN Gateways to automatically learn Spoke VNETs' prefixes and route traffic via NVA. Your actual configuration can be showed as this one:

The solution will require deploy an Azure Route Server in the HUB VNET, This enables dynamic route propagation and learning between components. Bellow configuration required:

Peer the NVA with the Azure Route Server to advertise learned routes.

Peer the VPN Gateway with the Azure Route Server to dynamically learn Spoke VNET prefixes.

On Spoke VNETs set a default route (0.0.0.0/0) pointing to the NVA for outgoing traffic

On HUB VNET GatewaySubnet include a UDR to send all traffic destined for on-premises to the NVA first, then to the VPN Gateway.

Therefore the Traffic Flow on Spoke VNETs send traffic to the NVA due to the default route and VPN Gateway learns Spoke VNET prefixes from the Route Server and routes traffic through the NVA, ensuring traffic inspection before it reaches its destination.

This configuration, eliminates the need for manual UDR updates for each prefix and ensures all traffic from Spoke VNETs is inspected by the NVA before reaching on-premises or other destinations.

References:

https://learn.microsoft.com/en-us/azure/route-server/peer-route-server-with-virtual-appliance

https://learn.microsoft.com/en-us/azure/route-server/route-server-faq

https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

Regards,

Luis
Cat Mucius 71 Reputation points

2024-10-21T10:59:02.6033333+00:00
Hi Luis,

Thank you for quick and detailed answer, but it begs more questions. :-)

The "Allow gateway or route server in <HUB-VNET> to forward traffic to the peered virtual network" & "Enable <Spoke-VNET> to use <HUB-VNET> remote gateway or route server" options, as their names suggest, affect both Route Server and VPN Gateway.

Hence, if I enable them for my VNET peerings, then how can I avoid the VPN Gateway learning Spoke VNETs ranges?

And if this cannot be avoided, then how can I force VPN Gateway to deliver VNETs-destined traffic to NVA - without overriding each and every such range in the UDR attached to the GatewaySubnet?

As I understand, I can peer my Spoke VNETs with HUB-VNET hosting the VPN Gateway using the AllowGatewayTransit & UseRemoteGateways flags, and still avoid Spokes sending traffic to on-premises narrow routes directly to the VPN Gateway, by attaching a Route Table with Disable BGP route propagation option and with generic UDRs pointing to my NVA to their subnets.

But how it can be done in the opposite directions, for packets delivered by the VPN Gateway to the Spokes?

"On HUB VNET GatewaySubnet include a UDR to send all traffic destined for on-premises to the NVA first, then to the VPN Gateway."

I don't understand this. Such UDR can be applied only to packets sourced from the subnet it's attached to - meaning GatewaySubnet, which in turn means the VPN Gateway, isn't it?

If I apply such a UDR to GatewaySubnet, then either it will be ignored (if the VPN Gateway has learned a narrower prefix from on-premises) or, if the prefixes are equal, I'll create a routing loop, with packets bouncing back & forth between the Gateway and the NVA.

Anyway, forcing Spokes-to-on-premises traffic to flow via the NVA is not a problem - the problem is with the opposite direction.

Thanks!

Mucius.
Cat Mucius 71 Reputation points

2024-10-22T07:29:49.91+00:00

Let's say VPN Gateway will learn Spoke VNETs ranges from NVA - by means of BGP peerings: VPN Gateway <====> Route Server <=====> NVA.

But then the question is: how the NVA can discover the VNETs' ranges?

(Update: apparently via Azure Route Server.)

Is it possible somehow to add a static route to the VPN Gateway's own routing table, saying something like deliver packets for 10.11.0.0/16 to the SDN (lowest IP of GatewaySubnet) ?
I mean not the UDR applying to packets already delivered by the Gateway to SDN - but the routing table inside the Gateway's instances.
Cat Mucius 71 Reputation points

2024-10-22T09:09:08.85+00:00

Just ran a test: peered a VNET hosting my VPN Gateway directly with a Spoke VNET, while enabling the AllowGatewayTransit & UseRemoteGateways flags for the peering and disabling Propagate gateway routes for the Spoke VNET's subnet hosting a VM.

Good thing: the routing table of VPN Gateway learned the Spoke's range, so it wouldn't be overriden by a wider prefix learned from elsewhere. The Gateway should deliver packets destined there to its GatewaySubnet according to this range, not as a last resort.

Bad thing: apparently, the peering caused the SDN to program NICs of the Gateway instances to know the Spoke's range too, so when the Gateway delivers packets to the SDN, they got routed straight to the destination VM, bypassing my NVA. (The reply packets, however, do flow to the NVA, which drops them).
The only way to avoid that, as I suspected, is to override the Spoke's prefix in the Route Table associated with GatewaySubnet.

Do you have any idea how fix that without configuring such overrides for tens or even hundreds of Spokes?
Ganesh Patapati 1,975 Reputation points Microsoft Vendor

2024-10-24T15:46:21.6966667+00:00

Hi Cat Mucius,

Greetings,

Welcome to the Microsoft Q&A Platform. Thank you for posting your query here. We are looking into it and will get back to you soon.

Thank you.
Ganesh Patapati 1,975 Reputation points Microsoft Vendor

2024-10-28T17:18:58.73+00:00
Hey Cat Mucius,

We appreciate your patience!

Is your issue addressed, or do you require further assistance on this thread?

Looking forward to your response and appreciate your time on this.

Regards,

Ganesh
Cat Mucius 71 Reputation points

2024-10-28T22:12:55.4833333+00:00

Hi Ganesh,

If you have any idea how I can fix my setup to make the firewalls work in active-active fashion (while avoiding the issues it came to fix - VPN Gateway not knowing any IP ranges behind it or need to maintain tens or potentially hundreds of UDRs) - I would be grateful. :-)

Thanks,

Mucius.
Ganesh Patapati 1,975 Reputation points Microsoft Vendor

2024-10-29T15:31:08.5966667+00:00
Hey @Cat Mucius

We Appreciate your patience!

Have you validated or tested this in a lower environment?

Do you have a working deployment

Because your configuration appears to be highly complex, and I will not be able to validate it by deploying it.

My recommendation would be to advertise a more specific route from NVA, additional VNET in between Spoke and Hub.

Keep the VPN gateway, the route server, and the NVA in the same vnet (Hub Vnet), and advertise the more specialized route from the NVA.

When Route Server has multiple routes to an on-premises destination prefix, Route Server selects the best route(s) in order of preference, as follows:

Prefer routes with the longest prefix match (LPM).

Prefer routes based on routing preference configuration.

Refer: https://learn.microsoft.com/en-us/azure/route-server/hub-routing-preference#routing-preference-configuration

I hope this clarifies, if not please let me know.

Regards,

Ganesh
Cat Mucius 71 Reputation points

2024-10-31T15:34:25.41+00:00

Thanks, Ganesh, but while advertising routes from NVA that are narrower than the Spokes' ranges is possible, this is no better than simply overriding them by UDRs associated with GatewaySubnet.

Since the maximal number of Spokes that can be attached to the Hub VNET is 500, I'm searching for solution that would allow us to avoid this kind of work.

Thanks!

Mucius.
Ganesh Patapati 1,975 Reputation points Microsoft Vendor

2024-11-01T18:31:08.2033333+00:00

Hey Cat Mucius,

We appreciate your patience!!

In this below thread is what you're trying to achieve:

Refer: https://learn.microsoft.com/en-us/answers/questions/1792054/route-on-prem-traffic-through-an-azure-nva-to-a-vp

Hope this clarifies,

Please let me know if your still facing issues.

Regards,

Ganesh
Ganesh Patapati 1,975 Reputation points Microsoft Vendor

2024-11-05T14:35:00.8+00:00

Hello Cat Mucius,

Good day!

I would like to follow up with the thread.

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If you are still facing any further issues, please don't hesitate to reach out to us. We are happy to assist you.

Looking forward to your response and appreciate your time on this.

Regards,

Ganesh
Ganesh Patapati 1,975 Reputation points Microsoft Vendor

2024-11-07T12:16:49.38+00:00
Hello Cat Mucius

Greetings,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I won't recommend you to be make more complex this scenario so,

My recommendation would be to advertise a more specific route from NVA, additional VNET in between Spoke and Hub.

Keep the VPN gateway, the route server, and the NVA in the same vnet (Hub Vnet), and advertise the more specialized route from the NVA.

When Route Server has multiple routes to an on-premises destination prefix, Route Server selects the best route(s) in order of preference, as follows:

Prefer routes with the longest prefix match (LPM).

Prefer routes based on routing preference configuration.

Refer: https://learn.microsoft.com/en-us/azure/route-server/hub-routing-preference#routing-preference-configuration

I hope this clarifies, if not please let me know.

Regards,

Ganesh

1 answer

Cat Mucius 71 Reputation points

2024-10-25T23:34:05.5166667+00:00
Figured out a partial solution:

The idea is this:

Since the best way to inject routes into VPN Gateway's internal routing table is by BGP, nothing prevents us from establishing BGP-over-IPsec peerings between the Gateway and our firewall VMs.

So the firewall VMs will dynamically learn Spoke VNETs' prefixes from Azure Route Server and re-advertise them to the VPN Gateway instances.

The VPN Gateway will be deployed in a separate VNET, to avoid its GatewaySubnet learning direct routes to Spokes.

The main problem of this solution is that it requires Active-Passive firewall setup, to keep traffic symmetry.

Documented more details here:

https://github.com/cat-mucius/kb/wiki/Dynamic-routing-with-Azure-VPN-Gateway-and-with-traffic-inspection-by-custom-firewall

Update: judging by the "Direct routing to NVA instances for dual-role connectivity and firewall NVAs" section in Virtual WAN documentation, Microsoft themselves didn't find a good active-active solution for this problem. :-)

Azure's software-defined networking platform doesn't guarantee flow-level symmetry, meaning the inbound flow to Azure and outbound flow from Azure can land on different instances of the NVA. This results in asymmetric routing which is dropped by stateful firewall inspection. Therefore, it isn't recommended to use active-active connectivity patterns where an NVA is behaving as a dual-role NVA unless the NVA can support asymmetric forwarding or support session sharing/synchronization.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Dynamic routing with VPN Gateway and with inspection by NVA

1 answer

Your answer