Troubleshooting Bad Requests Through a Private Endpoint Connected to a Load Balancer

Omer Cohen 40

Greetings,
I have a private endpoint connected to private link service composed of a load balancer with a VM in its backend. The load balancer's Inbound rule forwards TCP traffic on port 80 to the backend pool containing aforementioned VM. The backend VM forwards traffic through a VPN gateway to a remote service, using iptables. This is confirmed to work correctly: running curl <private-ip-of-remote-ip> from any VM in the VNet returns a valid response. However, when running curl <private-ip-of-load-balancer> I'm getting a bad request. Obviously the same occurs when running curl <ip-of-private-endpoint> (since the LB and private-endpoint are connected). When temporarily replacing the load balancer's inbound rule with a nat rule I was able to get a valid response when running curl <private-ip-of-load-balancer> from a VM in the load balancer's VNet, however, this is not a solution, since a private endpoint isn't compatible with an inbound nat rule. Attached is a diagram containing the resources and rules mentioned above:
Screenshot from 2024-11-18 20-49-20

I would greatly appreciate your expertise in the matter,
Omer.

1 answer

KapilAnanth-MSFT 47,996 Reputation points Microsoft Employee

2024-11-19T04:26:06.33+00:00

@Omer Cohen ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I believe this is a continuation of : https://learn.microsoft.com/en-us/answers/questions/2114387/exploring-cost-effective-solutions-for-routing-tra

We shall continue to work with you in the above thread as well.

It appears some part of your question has been removed, can you please add them back so I can have a better understanding of the issue.

Cheers,

Kapil
Please sign in to rate this answer.
Omer Cohen 40 Reputation points

2024-11-19T16:46:04.4966667+00:00

Hey ,
I've completed the part that wasn't uploaded correctly, although it seems that formatting is off.
This question raises a different issue, one that does not overlap with the others.

I would like to add that before including iptables on my approach, I've tried following this guide by enabling ip forwarding on the VM's nic and utilizing a routing table route.

Thanks,
omer.

KapilAnanth-MSFT 47,996 Reputation points Microsoft Employee

2024-11-20T03:52:15.3466667+00:00

Thanks @Omer Cohen , for the explanation.

From your verbatim,

When you say "nat rule", I take it that you are talking about inbound NAT rules.

Observation,

If you are using "curl", I can only assume that the "remote-ip" resource should be HTTP/HTTPS.

From our discussion from the other thread, this should be possible with App gateway without needing a intermediate VM

See : App Gw Components Backend pools

With that said,

The fact that the traffic works with Inbound NAT Rule and not with Load Balancer Rule indicates there is a misconfiguration.

Did you use Azure VM or Backend Pool in the Inbound NAT Rule?

Unlike LB Rules, Inbound NAT rules does not need a health probe

This means, if the backend is not responding to health probes, traffic would not be sent to the backend VM in case of a LB Rules

Can you please check "Data Path Availability" and "Health Probe Status" from the SLB metrics? Are they 100% ?

Are you using default probe or custom probes? Make sure the backend VM can respond to the health probes

Cheers,

Kapil

Omer Cohen 40 Reputation points

2024-11-20T16:46:04.78+00:00

Hey @KapilAnanth-MSFT ,

Correct, the remote server is a VM hosting an Apache server.

I would prefer a solution utilizing an application gateway, but per my last reply on the other post, I'm encountering a lot of bugs, so in the meanwhile I would like to progress on both fronts, due to time limitations.

The inbound NAT rule used a backend pool as a backend.

Data Path Availability and Health Probe Status both average at 100.

KapilAnanth-MSFT 47,996 Reputation points Microsoft Employee

2024-11-22T04:35:02.3933333+00:00

@Omer Cohen ,

The next steps would be,

When you say "bad request"

Is it HTTP 400 response code?

Or connection timed out OR similar errors?

Are you able to TcpPing / Telnet (instead of curl) to the LB's IP?

Does the connectivity work on Layer 4?

Or is it timing out or giving you any errors?

If TCP works and HTTP does not, please check the application logs and see why 400 was given out by the Apache server

However, if neither TCP nor HTTP works,

Please consider collecting a simultaneous packet captures at the sourceVM, backendVM (behind LB) and the remoteServer

This should give us an idea of where the packet is getting lost

Cheers,

Kapil

Omer Cohen 40 Reputation points

2024-11-24T17:50:33.7633333+00:00

Hey @KapilAnanth-MSFT ,

Is it HTTP 400 response code?

It is indeed HTTP 400 response code.

Are you able to TcpPing / Telnet (instead of curl) to the LB's IP?

telnet <load-balancer-ip> 80 instantly returns the following, pointing to an issue on the application layer:

Connected to 10.57.3.5.

Escape character is '^]'.

HTTP/1.1 400 Bad Request

content-length: 11

content-type: text/plain

date: Sun, 24 Nov 2024 16:57:30 GMT

via: 1.1 google

connection: close

While the following nc -zv <load-balancer-ip> 80 returns: Connection to <load-balancer-ip> 80 port [tcp/http] succeeded! indicating The TCP connection to port 80 is successfully established.

However, curl/telnet <remote-ip> returns the correct response via the following route:
VM->S2S VPN gateway -> private-endpoint -> Apache server.

Please consider collecting a simultaneous packet captures at the sourceVM, backendVM (behind LB) and the remoteServer

The Apache server's error log file doesn't contain records related to this matter.
I will update if I manage to find relevant records via tcpdump on either VM, but so far no luck there.

KapilAnanth-MSFT 47,996 Reputation points Microsoft Employee

2024-11-25T09:53:31.01+00:00

@Omer Cohen , Load Balancer is incapable of giving out a HTTP response error code (as it operates on L4).

I think the issue should be related to the VM(backend of the LB) or the apache server itself.

Sure, you can collect captures and let us know.

Cheers,

Kapil

Omer Cohen 40 Reputation points

2024-11-25T19:20:35.6866667+00:00

Hey @KapilAnanth-MSFT ,
While I keeping looking into the logs, I wonder whether I gave up on a solution that doesn't involve making major configuration changes in the router-VM's OS (which is presented in the first message's diagram) via iptables.

I'll reiterate the required workflow on Azure:
consumer-VM -> private-endpoint -> load-balancer - > router-VM -> VPN gateway -> ... -> remote service.
We have established that the load balancer might be misconfigured, since the following workflow does work:
vm-in-router-vm-vnet - > router-VM -> VPN gateway -> ... -> remote service.

I suspect that the iptables configuration on the router-VM, forwarding all incoming tcp:80 traffic to remote service's private IP, might not be compatible for the load balancer.
Can we somehow replace it? From this documentation and this Stackoverflow post , it seems that a UDR and IP-forwarding enablement on the VM's (virtual) NIC and OS might be all there is to it. I wasn't able to replicate it, but I would appreciate your expertise in the matter.

Many Thanks,
Omer.

KapilAnanth-MSFT 47,996 Reputation points Microsoft Employee

2024-11-26T05:54:51.26+00:00

@Omer Cohen ,

We have established that the load balancer might be misconfigured, since the following workflow does work: vm-in-router-vm-vnet - > router-VM -> VPN gateway -> ... -> remote service

I am afraid I cannot agree to this

As it's quite possible that the routerVM is not forwarding the packets to the "remote service" or not resending the packets back to the "vm-in-router-vm-vnet"

The correct way to figure this out is by collecting captures at the router-VM and see whether or not the packet is received by the router-VM, and did it forward the packets and receive a response from the "remote service" or not.

From Azure side, you are correct - Route Tables and Enabling IP forwarding on NIC is enough.

Wrt, "I suspect that the iptables configuration on the router-VM, forwarding all incoming tcp:80 traffic to remote service's private IP, might not be compatible for the load balancer."

I don't think so

One thing you must note here is that return traffic does not pass via the LB (Direct Server Return (DSR) )

What this means is once the LB sent the request traffic to the router-VM, LB's job is done

During the response traffic, router-VM should send it back to the "vm-in-router-vm-vnet" directly

So,

If we could do a packet capture and see the "request" traffic reaching the router-VM, I would say LB's job is done.

And if we see the "response" traffic reaching the router-VM, the flow is fine.

Hope this adds more clarity.

Cheers,

Kapil
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Troubleshooting Bad Requests Through a Private Endpoint Connected to a Load Balancer

1 answer

Your answer