Integrate Cisco SDWAN OnRamp for MultiCloud with a Cisco FTDv pair at the edge of the Azure environment

Eric G 5

Diagram

I am trying to add Azure to my Cisco SDWAN environment using Cisco's OnRamp for MultiCloud. I also need packets inspection (both directions) by Cisco FTDvs (pair) configured between Azure load balancers (LB). So ingress traffic should flow as such: vWAN -> vHub (created by Cisco SDWAN) -> Cisco FTD LB -> Cisco C8000v SDWAN routers -> Azure resources (egress, reverse, skipping C8000vs if Internet-bound; traversing the C8000vs if SDWAN-bound). I have tried multiple configuration adjustments without any success: manipulating route tables, adding route maps, iBGP between the FTDs and C8000vs, eBGP between the vHub and FTDs, etc., ad nauseam.When deploying the vWAN and vHub via Cisco OnRamp, it puts everything "physically" (for lack of a better descriptor) behind the vHub. The diagram attached shows the topology as created by Azure when deploying OnRamp via Cisco vManage. I feel like we're trying to force something that might not be possible even though route manipulation should solve this, especially considering we're dealing with all L3 devices (vHub, C8000vs, and FTDvs). Any recommendations would be welcomed!

Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-09-10T04:01:29.2633333+00:00

Hi Eric G,

Welcome to the Microsoft Q&A Platform. Thank you for posting your query here. We are looking into it and will get back to you soon.

Thanks,

Rohith Vinnakota
Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-09-10T06:10:21.91+00:00

Hi Eric G,
Can you please share more details on the issue so that we can assist you further on this.

Can you provide details on the specific configuration steps you’ve taken?

Thanks,
Rohith Vinnakota
Eric G 5 Reputation points

2024-09-10T14:19:43.4366667+00:00

Hi Rohith,

Thanks for helping! In the beginning, we were only passing SD-WAN (enterprise, private) traffic through the C8ks and Internet traffic (ingress and egress) through the FTDvs. At that time, we had joined all vNets to the vHub via Cisco vManage Cloud OnRamp configuration, including the vNet in which the FTDvs were housed. We created an iBGP connection between the C8ks and the FTDvs (both "inside"-facing interfaces), primarily to push the default route from the FTDvs to the C8ks to point all Internet-bound traffic to the Inside Load Balancer set in front of the FTDv pair. We got that working when we peered vNets in which test VMs are housed to the FTDv vNet and creating a route table for those vNets pointing 0.0.0.0/0 to the Inside Load Balancer of the FTDv.

Then, the security team came back and asked us to pass all traffic through the FTDvs, SD-WAN included. This is now where I am stuck. Not getting eyes into Azure-specific resources (load balancers, vHub, etc.), I am not 100% sure, but I think there is a route loop happening for Internet-bound traffic between the C8ks and the FTDvs when we try to push all ingress and egress traffic through the FTDvs. Based on the diagram created by Azure (in my original post above), the vHub technical sets as the default "router" for the SD-WAN environment. Because the FTDvs are technically joined to the SD-WAN environment, any 0.0.0.0/0 route is sent out of the FTDv's Outside-zone interface (route statically configured on the FTDvs) which I assume passes through the vHub; there is a public IP attached to each Outside-zone interface. From there, I can only assume that the 0.0.0.0/0 in the SD-WAN environment within that Azure region points back to the Inside Load Balancer of the FTDvs and through the Inside-zone interface of the FTDvs while SD-WAN traffic routes through the vHub.

I have tried removing iBGP between the C8ks and FTDvs and just manipulating the vHub route table; adding route tables to the vHub (that broke SD-WAN connectivity); joining the FTDvs to the vHub as a eBGP connection; manipulating route tables for test VMs. I can get SD-WAN traffic flowing as expected when a default route is not configured on the vHub, but the VMs and FTDvs are not able to reach Internet destinations.

Let me know if any additional information would be helpful.
Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-09-11T09:51:14.0066667+00:00

Hi Eric G,
Thank you for getting back

Check Your Route Tables:

User Defined Routes (UDRs) are the custom routes for management of the traffic. You will have to ensure that they are leading the traffic to FTDv devices so when the traffic is moving via SDWAN to the internet, the router is being skipped.

Azure Load Balancer Setup:

As you are using a pair of FTDv devices on your network, you have to make a balance in your Azure Load Balancer. Make sure that the balancer is configured to distribute the load equally between them. Besides this, the verification of health probes is required so that the load balancer can learn which FTDv is healthy and traffic can be sent correctly.

Cisco OnRamp is a software that automatically sets up a network for a client, but sometimes, it is the end user who will have to labor in the Cisco vManage system just to do minor adjustments and set up new paths and policy settings.

Route Maps for Traffic Flow:

To traffic that goes exactly where you want it to go, use route maps together with Azure UDRs. In this manner, you can make it sure that only the specific traffic flows (like incoming traffic) go via the FTDv devices, and at the same time block the internet traffics that take a straight path

I know this setup can get tricky, but with these adjustments, you should be able to get things working smoothly. Feel free to reach out if you have more questions or need further guidance.

Thanks,
Vinnakota Rohith
Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-09-12T16:59:05.3233333+00:00

Hi Eric G,

Just checking in to see if you had a chance to see my response to your question. Please tell us if it was helpful and feel free to reach out to us if you have any queries.

Thanks,

Rohith
Eric G 5 Reputation points

2024-09-12T18:02:41.15+00:00

Hi Rohith,

I haven't had a chance - I've been tied up with other projects this week. In my environment, I do have full control of Azure Portal and Cisco vManage (for Cisco OnRamp), so I will double check everything according to your comment above and add any additional configurations (UDRs, Route Maps, etc.).

The load balancers are deployed as default, so I assume round-robin. The FTDvs were deployed in an availability set and send heartbeats to the LBs. Again, I will double check the LB configuration.

I will let you know if I have any progress or have any additional questions. Thanks again, Rohith, for your suggestions!

Eric
Eric G 5 Reputation points

2024-09-12T18:29:35.56+00:00

Hi Rohith,

I haven't had a chance to take a look at this due to other projects. I have full control of our Azure Portal and Cisco vManage, so I will be able to look at UDRs, Route Maps, etc. per your suggestions.

The FTDvs were deployed in an availability set between the Load Balancers and do have heartbeats to each LB (inside and outside). The LBs are configured as default, so I assume round-robin between the FTDvs with the assumption that, because the FTDs egress (internal or Internet) directly to destinations, the destination returns any packets to the FTD that formed the session, not the LB.

I will reference your comments and reach out if I have any further questions. Thanks again for your responses and suggestions!

Eric
Eric G 5 Reputation points

2024-09-17T18:09:39.08+00:00

Hi Rohith,

I am working on this today. I have been curious about the Outside Load Balancer (OLB) load balancing rule. The guide I have been using was specific to web server access. However, we are looking for any access, so any port. The rule (included screenshot) only allows for a single port. Is a rule required for each port that may enter through the OLB, or is this only for ingress traffic sessions that start from outside (like a web server)?

*I set Session persistence to "Client IP and protocol" so once a session is passed through a LB to the FTDs, the FTDs then maintain that session for consistency. The same is configured on the Inside LB.
Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-09-18T00:54:43.5966667+00:00

Hi Eric G,

Good Day!

Yes, you can create a separate rule for each port that you want to allow incoming traffic.

*I set Session persistence to "Client IP and protocol" so once a session is passed through a LB to the FTDs, the FTDs then maintain that session for consistency. The same is configured on the Inside LB.

answer: yes
Thanks,
Rohith
Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-09-19T00:58:23.86+00:00

Hi Eric G,

Greetings of the day!!

Just checking in to see if you had a chance to see my response to your question. Please tell us if it was helpful and feel free to reach out to us if you have any queries.

Thanks,

Rohith
Eric G 5 Reputation points

2024-09-23T15:07:30.6433333+00:00
Hi Rohith,

I have read through your responses and tried a few things related to your suggestions, but I have not had much luck. We started with this basic topology:

Interfaces for reference:

C8000v Inside: x.x.x.229 and x.x.x.230

C8000v Outside: x.x.x.244 and x.x.x.245

FTDv Inside: x.y.x.4 and x.y.x.5 (x.y.x.68 for the single FTDv)

FTDv Outside: x.y.x.20 and x.y.x.21 (x.y.x.84 for the singe FTDv)

Inside Load Balancer: x.y.x.10

The vWAN, vHub, and SDWAN NVAs (C8000v) were deployed through Cisco vManage using OnRamp for Multicloud. The test-vNets and Transit-FTD-vNet are peered to the vHub/SDWAN routers through vManage configuration. This configuration appears to function for connectivity (SDWAN environment and test VMs egress traffic to the Internet). However, SDWAN traffic does not flow through the FTDvs, which is required; only Internet-destined/sourced traffic flows through the FTDvs.

The FTDvs were deployed two different ways for testing:

(above diagram) As a pair in an availability set (AS) between an Inside Load Balancer and Outside Load Balancer (ILB/OLB) (as previously described above in this thread).

For testing purposes, I simplified the topology by just deploying a single FTDv without the AS or ILB/OLB (topology diagram further below in this comment).

The test-vNets (100 and 200) both are associated to a UDR pointing their 0.0.0.0/0 route to the FTDv (either the ILB - x.y.x.10 - for option 1 above or the inside interface of the single FTDv - x.y.x.68 - for option 2).

For routing, I configured iBGP between the inside interfaces of the C8000vs and FTDvs and added route maps to their iBGP configurations. From the NVAs' perspective (verified via CLI), they formed an iBGP neighborship and were sharing routes: the C8000vs advertised SDWAN routes, the FTDs advertised the test-vNets to the C8000vs. Using a route map, I suppressed the C8000vs from advertising the test-vNet routes to the vHub, assuming the C8000vs and FTDvs could manage the routing for the vNets, without the vHub, using their iBGP connection. The FTDvs advertised a 0.0.0.0/0 route to the C8000vs; on the C8000vs; I added a route map and prefix list to change the default route to the ILB. I decided not to edit the Route Table of the vHub; every time I attempted to add UDRs on that Route Table, connectivity (internal/SDWAN) to the FTDvs was lost.

When SDWAN traffic failed to flow through the FTDvs, I experimented with this topology:

In this configuration, I removed the test-vNets from the SDWAN environment, as configured in vManage, and peered them with the Transit vNet (FTDv vNet). The FTDvs had no problem reaching the test VMs. However, SDWAN had no connectivity to the test-vNets. The C8000vs still received the routes to the test-vNets from the FTDvs via iBGP. The only routing change made was the prefix list on the C8000vs for the 0.0.0.0/0: it pointed to the inside interface of the single FTDv instead of the ILB used for the initial topology. I ran a traceroute from my MacBook (connected to our SDWAN environment) to the test VMs; the packets bounced between the inside interfaces (x.x.x.229 and x.x.x.230) of the C8000vs, never getting to the FTDvs or the test-vNets.

I did experiment with eBGP directly between the vHub and Transit-FTD-vNet; that, however, never formed a neighborship from the FTDvs' perspective. Based on that failure, I assume that configuring BGP peering in a vHub peers to the vNet itself, not NVAs within that vNet.

The only thing I can think of is with this environment: in both topologies, the vHub and/or any vNet/subnet gateways managed by Azure are throwing off the required routing flow (all traffic, SDWAN and Internet-bound/sourced, flowing through the FTDvs). Route Maps are available as a preview for vHubs, but they are (1) not recommended for production yet; and (2) are limited compared to route maps that we can configure directly on our Layer 3 NVAs (C8000vs and FTDvs). Are there any Azure resources that need to be added/included in order to accomplish the traffic flow that we are required (VPN Gateway, additional UDRs, etc.)?
Eric G 5 Reputation points

2024-09-23T15:19:04.1666667+00:00

Hi Rohith,

I thought the above comment failed to post, so please ignore this comment :)
Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-09-24T04:42:46.8066667+00:00

Hi Eric G,
Good day!

Thank you for getting back
you can use the route intent and route policy to accomplish the traffic flow. Here is the complete detail about this: How to configure Virtual WAN Hub routing policies - Azure Virtual WAN | Microsoft Learn

I hope this will help you

If you have any further concerns, please do not hesitate to contact us.

We are pleased to help you.

Thanks,

Rohith
Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-09-25T23:07:32.8366667+00:00

Hi Eric G,

Greetings of the day!!

Just checking in to see if you had a chance to see my response to your question. Please tell us if it was helpful and feel free to reach out to us if you have any queries.

Thanks,

Rohith
Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-09-26T22:44:24.6266667+00:00

Hi Eric G,

Greetings of the day!!

Just checking in to see if you had a chance to see my response to your question. Please tell us if it was helpful and feel free to reach out to us if you have any queries.

Thanks,

Rohith
Eric G 5 Reputation points

2024-10-01T15:18:51.7266667+00:00

Hi Rohith,

Some good news - we have routing working as we want it using a mix of BGP peering and route maps. The VMs are peered directly to the Transit-vNet (FTDv, single without load balancers for testing purposes), and the FTDv is peered over eBGP to the vHub. All Internet-bound and SDWAN-bound traffic is flowing from the VMs through the FTDv. We are using the following topology:

We do have a major issue: The VMs can send and receive ICMP and we can SSH into them. However, no other services are working (VM SSH out, file transfer in/out, 80/443 out, etc.). DNS only works with Azure DNS servers and not our internal DNS server.

All NSGs have allow any/any/any (source/destination/port) as the top rules for both directions on all devices (VMs and FTDv outside interface). The FTDv Access Control Policy is configured with a single allow any/any/any rule. The firewalls on the VMs are disabled. As far as access control is concerned, all rules allow all traffic to/from anywhere over any port.

Can you think of anything in Azure's infrastructure that would block most services and only allow ICMP in/out and SSH in? This is a huge hurdle that we need to overcome in order to finalize our testing before deployment.

Thanks again for all of your suggestions and recommendations!

Eric
Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-10-01T20:33:58.9066667+00:00

Hi Eric G,

Good day!

I’m glad to have solved that problem.

Once again check the NSG and firewall settings and route table

Ensure that the subnet has enough IP addresses available for all the VMs. IP exhaustion can lead to unpredictable connectivity issues, including the failure of certain services.

Use Azure Network Watcher to test the connectivity between the VMs and external endpoints
Thanks,
Rohith
Eric G 5 Reputation points

2024-10-02T15:41:05.51+00:00

Hi Rohith,

Just to clarify: All of my NSGs allow any/any over any port, so no rules on the NSGs are blocking anything. My VM vNets only have 1-2 VMs each with a /24 subnet for each vNet, so plenty of IPs. Packet captures from the FTDvs show traffic and NATing are functioning as expected. At this point, I can only see issues on the Azure side with the Public IP attached to the FTDvs' outside vNIC.

Can you think of any reason why the VMs can ICMP in/out, I can SSH (and SFTP) and RDP in to the VMs, but no other services out from the VMs are working (VMs cannot SSH out to each other or other VMs, they cannot reach out to websites over 80/443, and cannot file transfer out over FTP/SFTP/TFTP). Based on all NSGs and Access Control Policies in the FTDvs, they should have full access out and any device that can reach the VMs should have full access in.
Eric G 5 Reputation points

2024-10-04T15:28:41.9433333+00:00

Hi Rohith,

Quick update - we have everything 100% working as we need it. We had to make a few minor changes to routing and NAT, but everything is working. Appreciate your assistance and suggestions!

Thanks,

Eric

1 answer

Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-10-07T16:49:02.9066667+00:00

Hi Eric G

Good day!

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Issue: Integrate Cisco SDWAN OnRamp for MultiCloud with a Cisco FTDv pair at the edge of the Azure environment

Solution: we have routing working as we want it using a mix of BGP peering and route maps. The VMs are peered directly to the Transit-vNet (FTDv, single without load balancers for testing purposes), and the FTDv is peered over eBGP to the vHub. All Internet-bound and SDWAN-bound traffic is flowing from the VMs through the FTDv. We are using the following topology:

Thank you again for your time and patience throughout this issue.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.
Please sign in to rate this answer.

1 person found this answer helpful.
Eric G 5 Reputation points

2024-10-07T22:30:42.37+00:00

There are some additional configurations (route maps and prefix lists) that need to be configured on the FTDvs and C8000vs. If anyone has a similar topology, feel free to reach out to me regarding those configurations.

Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-10-07T22:41:05.2033333+00:00

Hi Eric G,

Select the "Accept answer" button underneath the answer.

The accepted answer has a green frame to visually indicate that it's the accepted answer and the "Accepted answer" label.

Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-10-08T19:57:03.1266667+00:00

Hi Eric G,

Select the "Accept answer" button underneath the answer.

The accepted answer has a green frame to visually indicate that it's the accepted answer and the "Accepted answer" label.

Alex Price 0 Reputation points

2024-10-22T21:26:32.5466667+00:00

Hey Eric G, great job to find that final. What were the final route maps and prefix lists that were added? Is there a contact that you can share? Thanks.

Alex Price 0 Reputation points

2024-10-22T21:30:52.5166667+00:00

Hi Eric, great job to get it working. What were the final route maps and prefix lists? How to get in contact for details? Thanks.

Eric G 5 Reputation points

2024-10-24T13:07:40.36+00:00

Hi Alex,

C8000v (SDWAN): We used a CLI template for the route maps on the C8000v (C8k).

For eBGP between the C8ks and the vHubs, we added a prefix list for the Azure resources vNets and denied that as part of the default out-bound route map created during the Cloud OnRamp for Multicloud process. OnRamp manages the rest of that route map.

For eBGP between C8ks and FTDvs, we had two route maps. On the in-bound route map, we denied a prefix list that I'm questioning why (it's been a few weeks, and the environment is currently offline since it was primarily up for initial testing). For the out-bound route map, we created a prefix list that included the vNet in which the FTDvs live as well as the subnet for the C8ks, denying that. We also denied the Azure resource vNets as part of this same route map (this prevents the C8ks from advertising the vNet routes it learned from the vHubs back to the FTDvs, who already advertised those routes through their eBGP with the vHubs). Final route map statement, permit any.

FTDv:

For eBGP between both the C8ks and vHubs, we created a prefix lists that included all of our SDWAN hub/branch site /16 subnets (aggregated them to the per-site /16 in vManage as well) and denied that in an out-bound route map to both the C8ks and vHubs (with a default permit any at the end of the route map).

From this project, I've learned that trying to figure out routing in Azure with multiple Layer 3 NVAs is like trying to figure out a college-level theoretical calculus problem. In the end, as Doc Brown says, you have to think 4th dimensionally. Hope this helps! Let me know if you have any additional questions.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Integrate Cisco SDWAN OnRamp for MultiCloud with a Cisco FTDv pair at the edge of the Azure environment

1 answer

Your answer