ACL's blocking traffic to A VPN Gateway

Magalhaes César 31

Hi,

I've been working around a problem with a VPN Gateway i built a few days ago. It appears the traffic is blocked inbound to this gateway by an ACL that i cant edit, control or even see.

The former architecture is a hub/spoke design connected with an onPrem firewall.

At first i detected that i couldnt access the datacenter using ports other than 179, 443 and 500 (and a few more above 8800).

In order to troubleshoot this behavior, i remove parts of this architecture to isolate the problem.

So the setup now, is 2 vpn gateways connected via a site to site connection. My tests are done while placing 2 VM's in a different subnet, but on the same VNET of each of the gateways.

There a no NSG's configured in any of those subnets.

VNET A ( VM_A 172.18.0.100 + VNET GATEWAY A ) <—> S2S connection <—> VNET B ( VM_B 10.200.1.4 + VNET GATEWAY B )

Symptoms are :

If i do a port scan (netcat/nmap) from VM A to VM B, only 3 ports are observed to pass through vnet gateway A, and reach VM B (tcpdump 'host

Magalhaes César 31 Reputation points

2024-10-14T06:44:13.9366667+00:00

What the hell happened to this post, it's been cut in half...
Ganesh Patapati 1,745 Reputation points Microsoft Vendor

2024-10-14T10:04:23.3+00:00
Hello Magalhaes César,

Greetings!

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

Could you post topology for this and if applicable some configuration.

With regards,

Ganesh
Magalhaes César 31 Reputation points

2024-10-14T10:07:41.87+00:00

All the full description was available on the initial post. 80% of the post was truncated this morning for whatever reason.

I will post an answer with a new detailed analysis.

Magalhaes César 31

Analysis : The initial issue was reported as DNS traffic blocked from Azure to onPrem using this vnet gateway. But it appears most of the traffic is blocked.

Only 179, 443 and 500 ports are allowed and some above 8000

I reproduced the problem with a setup like this one :

VNET A ( VM_A 172.18.0.100 + VNET GATEWAY A ) <—> S2S connection <—> VNET B ( VM_B 10.200.1.4 + VNET GATEWAY B )

There is no NSG on any of the subnets on both VNET's.

The symptom is :

If i do a port scan (netcat/nmap) from VM A to VM B, only 3 ports are observed to pass through vnet gateway A, and reach VM B (tcpdump 'host <ip_vm_a>')
If i do the same test on the other way, VM B to VM A, i can reach VM B on any port.

VM_A

$ ip a s
172.18.0.100/27
$ nmap -p 21-1000 10.200.1.4
Starting Nmap 7.80 ( https://nmap.org ) at 2024-10-10 05:38 UTC
Nmap scan report for 10.200.1.4
Host is up (0.0054s latency).
Not shown: 977 filtered ports
PORT    STATE  SERVICE
179/tcp closed bgp
443/tcp open   https
500/tcp closed isakmp
Nmap done: 1 IP address (1 host up) scanned in 4.66 seconds
$ curl http://10.200.1.4  # there's an nginx running on both hosts, listening on 80)
^C
$ curl http://10.200.1.4
^C`

VM_B

$ ip a s
inet 10.200.1.4/28 metric 100 brd 10.200.1.15 scope global eth0
$ tcpdump 'net 172.18.0.0/23' 
05:38:28.000028 IP 172.18.0.100.52436 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [S], seq 980896241, win 64240, options [mss 1310,sackOK,TS val 1034077713 ecr 0,nop,wscale 7], length 0
05:38:28.000103 IP vm-temp-cesar-2.internal.cloudapp.net.https > 172.18.0.100.52436: Flags [S.], seq 2062150851, ack 980896242, win 65160, options [mss 1460,sackOK,TS val 4019323481 ecr 1034077713,nop,wscale 7], length 0
05:38:28.002681 IP 172.18.0.100.52436 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 1034077717 ecr 4019323481], length 0
05:38:28.002681 IP 172.18.0.100.52436 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [R.], seq 1, ack 1, win 502, options [nop,nop,TS val 1034077717 ecr 4019323481], length 0
05:38:29.210718 IP 172.18.0.100.52444 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [S], seq 565377449, win 64240, options [mss 1310,sackOK,TS val 1034078925 ecr 0,nop,wscale 7], length 0
05:38:29.210768 IP vm-temp-cesar-2.internal.cloudapp.net.https > 172.18.0.100.52444: Flags [S.], seq 40789260, ack 565377450, win 65160, options [mss 1460,sackOK,TS val 4019324692 ecr 1034078925,nop,wscale 7], length 0
05:38:29.213014 IP 172.18.0.100.52444 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 1034078928 ecr 4019324692], length 0
05:38:29.213014 IP 172.18.0.100.52444 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [R.], seq 1, ack 1, win 502, options [nop,nop,TS val 1034078928 ecr 4019324692], length 0
05:38:30.511161 IP 172.18.0.100.52456 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [S], seq 863036778, win 64240, options [mss 1310,sackOK,TS val 1034080226 ecr 0,nop,wscale 7], length 0
05:38:30.511215 IP vm-temp-cesar-2.internal.cloudapp.net.https > 172.18.0.100.52456: Flags [S.], seq 2062940104, ack 863036779, win 65160, options [mss 1460,sackOK,TS val 4019325992 ecr 1034080226,nop,wscale 7], length 0
05:38:30.513980 IP 172.18.0.100.52456 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 1034080228 ecr 4019325992], length 0
05:38:30.513980 IP 172.18.0.100.52456 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [R.], seq 1, ack 1, win 502, options [nop,nop,TS val 1034080228 ecr 4019325992], length 0
05:38:30.714218 IP 172.18.0.100.33590 > vm-temp-cesar-2.internal.cloudapp.net.500: Flags [S], seq 1130711786, win 64240, options [mss 1310,sackOK,TS val 1034080429 ecr 0,nop,wscale 7], length 0
05:38:30.714278 IP vm-temp-cesar-2.internal.cloudapp.net.500 > 172.18.0.100.33590: Flags [R.], seq 0, ack 1130711787, win 0, length 0
05:38:32.012037 IP 172.18.0.100.52466 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [S], seq 1185295426, win 64240, options [mss 1310,sackOK,TS val 1034081726 ecr 0,nop,wscale 7], length 0
05:38:32.012107 IP vm-temp-cesar-2.internal.cloudapp.net.https > 172.18.0.100.52466: Flags [S.], seq 389755755, ack 1185295427, win 65160, options [mss 1460,sackOK,TS val 4019327493 ecr 1034081726,nop,wscale 7], length 0
05:38:32.014634 IP 172.18.0.100.52466 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 1034081730 ecr 4019327493], length 0
05:38:32.014634 IP 172.18.0.100.52466 > vm-temp-cesar-2.internal.cloudapp.net.https: Flags [R.], seq 1, ack 1, win 502, options [nop,nop,TS val 1034081730 ecr 4019327493], length 0
05:38:32.132445 IP 172.18.0.100.42976 > vm-temp-cesar-2.internal.cloudapp.net.bgp: Flags [S], seq 3316973351, win 64240, options [mss 1310,sackOK,TS val 1034081831 ecr 0,nop,wscale 7], length 0
05:38:32.132483 IP vm-temp-cesar-2.internal.cloudapp.net.bgp > 172.18.0.100.42976: Flags [R.], seq 0, ack 3316973352, win 0, length 0

Like i said, if i do the same test on the other direction, the tcpdump for VM_A is spammed.

==> I then enabled the VNET Flow Logs

And while doing http request using these 2 vm's, we can observe the following:

VM_A -> http 80 requests -> VM_B

 {
"aclID": "69dadfbb-18eb-43ec-adfb-6ee8c21307da",
"flowGroups": [
  {
"rule": "DefaultInboundDenyAll",
"flowTuples": [
  "1728551909443,172.18.0.100,10.200.1.4,40486,80,6,I,D,NX,0,0,0,0",
  "1728551910451,172.18.0.100,10.200.1.4,40486,80,6,I,D,NX,0,0,0,0",
  "1728551911480,172.18.0.100,10.200.1.4,40486,80,6,I,D,NX,0,0,0,0",
  "1728551912499,172.18.0.100,10.200.1.4,40486,80,6,I,D,NX,0,0,0,0",
  "1728551913527,172.18.0.100,10.200.1.4,40486,80,6,I,D,NX,0,0,0,0",
  "1728551914552,172.18.0.100,10.200.1.4,40486,80,6,I,D,NX,0,0,0,0",
  "1728551916567,172.18.0.100,10.200.1.4,40486,80,6,I,D,NX,0,0,0,0",
  "1728551920696,172.18.0.100,10.200.1.4,40486,80,6,I,D,NX,0,0,0,0"
]
  }
]

This seems to be cause of the behavior. This flow log matches the mac address used by the vnet gateway A (all of the outbound traffic has the vnet gateway private ip address as source)

Is this normal to have this effective rule on the GatewaySubnet or GatewayNic Interface ?

And the other way around, this rule is matched for an equivalent traffic on the GatewaySubnet of the peer vnet gateway B.

   {
    "aclID": "aa28b221-acda-4709-aac5-66f061f12f8c",
    "flowGroups": [
      {
        "rule": "DefaultRule_AllowVnetInBound",
        "flowTuples": [
          "1728560065005,10.200.1.4,172.18.0.100,36458,80,6,I,B,NX,0,0,0,0",
          "1728560066858,10.200.1.4,172.18.0.100,36468,80,6,I,B,NX,0,0,0,0",
          "1728560070066,10.200.1.4,172.18.0.100,36478,80,6,I,B,NX,0,0,0,0",
          "1728560070633,10.200.1.4,172.18.0.100,36458,80,6,I,E,NX,1,158,1,110",
          "1728560072525,10.200.1.4,172.18.0.100,36468,80,6,I,E,NX,1,158,1,110",
          "1728560075645,10.200.1.4,172.18.0.100,36478,80,6,I,E,NX,1,158,1,110"
        ]
      }
    ]
  },

The ACL deployed on the GatewaySubnet of vnet gateway A is denying traffic, but i cant figure out why we have this ACL deployed, and why it is not the same as VNET Gateway B...

Ganesh Patapati 1,745 Reputation points Microsoft Vendor

2024-10-14T10:35:03.2566667+00:00
Hello Magalhaes César,

first let me know, did you actually deploy a vnet gateway VPN or NVA as a third-party VPN?

If it is a Regular VPN then Please Refer to the below document:

Refer: https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpn-gateway-settings#gwsub

Nsg on the gateway subnet are not recommended, so don't use it.

Regards,

Ganesh
Magalhaes César 31 Reputation points

2024-10-14T10:37:06.9266667+00:00

There's none.
Magalhaes César 31 Reputation points

2024-10-14T10:38:45.5033333+00:00

There are no NSG's. Plus, as far as i know, it is impossible to bind a NSG to a GatewaySubnet as it is not supported.
Ganesh Patapati 1,745 Reputation points Microsoft Vendor

2024-10-15T16:36:31.8133333+00:00
Hey Magalhaes César,

We appreciate your patience!

Statement: There are no NSG's. Plus, as far as i know, it is impossible to bind a NSG to a GatewaySubnet as it is not supported.

So, from the above statement what you want to achieve what is your exact requirement can you brief me in detail what you are trying to achieve.

Please let me know what you know by ACL as per our end ACL is either Network security group or Virtual network manager mention in the below document.

Refer: https://learn.microsoft.com/en-us/azure/network-watcher/vnet-flow-logs-overview#log-format

aclID: Identifier of the resource that's evaluating traffic, either a network security group or Virtual Network Manager. For traffic that's denied because of encryption, this value is unspecified.

If there are no NSG in between the resources no traffic will be blocked, Check the Local level OS Firewall where it is blocking.

Do not hesitate to contact us if you are still experiencing issues. We are happy to help you.

I value your time and look forward to hearing from you on this.

With regards,

Ganesh
Ganesh Patapati 1,745 Reputation points Microsoft Vendor

2024-10-16T15:10:21.76+00:00
Hello Magalhaes César,

Good day!

I simply wished to inquire if you have had a chance to review my response to how to resolve the issue based on your request.

If you find you are having more issues than we've mentioned here, please do not hesitate to contact us. We are pleased to help you.

Forward to your response and appreciate your time on this.

Regards,

Ganesh
Ganesh Patapati 1,745 Reputation points Microsoft Vendor

2024-10-17T11:13:39.3033333+00:00
Hello Magalhaes César,

Greetings!

I simply wished to inquire if you have had a chance to review my response to how to resolve the issue based on your request.

If you find you are having more issues than we've mentioned here, please do not hesitate to contact us. We are pleased to help you.

Forward to your response and appreciate your time on this.

Regards,

Ganesh
Magalhaes César 31 Reputation points

2024-10-17T16:39:23.8933333+00:00

Hi mate,

Host OS is irrelevant for this case. Because we see the denied traffic on a ACL hitting the VNET Gateway interface.

I have a support ticket opened for this issue, but it is taking ages.

I redeployed the VNET Gateway with the blocking ACL's to see if the redeployment could kind of reset the network layer. But the result is the same.

On my latest tests i have found an even weirder behavior. I use nmap to scan the port availability.

If i don't disable the host discovery (nmap ~~-Pn~~), i can see some traffic hitting for port 80 on the target. (I see some traffic with destination port 80 allowed on the VNET Flow logs)

If i disable the host discovery (nmap -Pn), nmap cannot mark port 80 as opened. (The FLow Logs show all traffic with same destination port 80 as denied ....)

I have an nginx listening on port 80 and 443 on the target, i can succesfully curl for the web service on 443 but not on 80.

The above behavior being even more confusing, i deployed a windows server, to make sure it is not a OS related problem, and i can reproduce this behavior, i can only reach the target with the 443 port. All the traffic outside 179, 443 and port 500 gets denied by the same ACL deployed on the GatewaySubnet or GatewayNIC.

This should be checked by the connectivity team, im out of options, and ideas.
Magalhaes César 31 Reputation points

2024-10-19T11:52:38.22+00:00

The reason of the problem is :

This is a hub & spoke design, so the GatewaySubnet has a route table to force traffic coming from vpn peers to spokes to be forwarded to the firewall.

This is mandatory, because if this route table doesnt exist, the traffic from vpn peers would use then the vnet peerings to reach the spokes. you would create asymetric routing, because the spokes have a default route to the firewall.

The spokes route tables have BGP Propagation disabled, if not, the system routes would be prioritary compared to the UDR's.

Because BGP Propagation is disabled per default in our case (all route tables are mostly applied on spokes), this setting was also disabled for the GatewaySubnet route table.

And ....This is the problem. Having this setting checked or unchecked does not have only a routing impact, but an ACL one.

This is mentioned in the documentation :

https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview#border-gateway-protocol

But this being counter intuitive, i indeed never suspected that.

I actually find abnormal that it partially works when BGP propagation disabled on the GatewaySubnet,

But that's it...
Ganesh Patapati 1,745 Reputation points Microsoft Vendor

2024-10-21T17:36:23.9366667+00:00
Hey Magalhaes César,

We appreciate your patience!

The recommendation is to work with azure support only because they have the In-detail details they have a right set of data to access and resolve your query better to follow-up with the support engineer whoever is contact with you.

With regards,

Ganesh

Accepted answer

Ganesh Patapati 1,745 Reputation points Microsoft Vendor

2024-10-29T11:37:23.6+00:00

Hey Magalhaes César,

Greetings,

Thank you for reaching out to us on the Microsoft Q&A forum.

As an original poster cannot accept their own answer, I am reposting it so that you can accept it an answer. Accepted answer will help other community members navigate to the appropriate solutions.

Issue: ACL's blocking traffic to A VPN Gateway

Solution: The reason of the problem is:

This is a hub & spoke design, so the GatewaySubnet has a route table to force traffic coming from vpn peers to spokes to be forwarded to the firewall.

This is mandatory, because if this route table doesnt exist, the traffic from vpn peers would use then the vnet peerings to reach the spokes. you would create asymetric routing, because the spokes have a default route to the firewall.

The spokes route tables have BGP Propagation disabled, if not, the system routes would be prioritary compared to the UDR's.

Because BGP Propagation is disabled per default in our case (all route tables are mostly applied on spokes), this setting was also disabled for the GatewaySubnet route table.

And .... This is the problem. Having this setting checked or unchecked does not have only a routing impact, but an ACL one.

This is mentioned in the documentation:

https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview#border-gateway-protocol

But this being counter intuitive, i indeed never suspected that.

I actually find abnormal that it partially works when BGP propagation disabled on the GatewaySubnet,

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information helps you, as this can be beneficial to other community members.

Your contribution is greatly appreciated.

Regards,

Ganesh
Please sign in to rate this answer.

1 person found this answer helpful.
Ganesh Patapati 1,745 Reputation points Microsoft Vendor

2024-10-29T11:39:17.7833333+00:00

Hello Magalhaes César,

We appreciate your patience!

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information helps you, as this can be beneficial to other community members.

Your contribution is greatly appreciated.

Regards,

Ganesh

Ganesh Patapati 1,745 Reputation points Microsoft Vendor

2024-10-30T14:52:04.6066667+00:00

Hello Magalhaes César,

We appreciate your patience!

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information helps you, as this can be beneficial to other community members.

Your contribution is greatly appreciated.

Regards,

Ganesh

Antoine 0 Reputation points

2024-11-18T21:24:12.4166667+00:00

Hello,

I have actually the same issue, some specifics port are blocked by the VPN GATEWAY NIC.

after activate Network watcher and checked the log, I found out the default ACL rule which deny some uncommon ports.

i checked the link you shared but

I don’t See any topic about acl of vpn gateway on this documents

thanks you for your help

Magalhaes César 31 Reputation points

2024-11-19T05:13:55.9833333+00:00

If that's the same "issue" : Check the routing table on the GatewaySubnet, make sure BGP propagation is not disabled.

Antoine 0 Reputation points

2024-11-19T09:51:43.65+00:00

should I configure BGP protocol also on the VPN gateway?

Magalhaes César 31 Reputation points

2024-11-19T11:00:32.35+00:00

Enabling BGP protocol on the VPN Gateway depends on the routing method you desire between your VPN peers and your gateway.

Between You Peered VNETs, i guess there's BGP behind the scenes like always. You just need to check the subnet on which the VPN Gateway is attached (GatewaySubnet), if it has a routing table, check the routing propagation property is enabled. And revalidate the behavior you observed
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

ACL's blocking traffic to A VPN Gateway

0 additional answers

Your answer