Cannot connect to Application Service from the Application Gateway when Private endpoints and Virtual Network Integration

Morgan Ecklund 0

Application Gateway give 502 error

Vidya Viraktamath 550 Reputation points Microsoft Employee

2025-02-16T18:05:23.7266667+00:00

Hi Morgan, Welcome to the Microsoft Q&A platform!

Ensure NSGs allow traffic between the App Gateway and backend, confirm UDRs are correctly configured, ensure DNS can resolve backend pool members' FQDNs, verify health probes are correctly set up and responding, check private endpoints are configured and connected properly, enable diagnostics and check logs for 502 errors.

You may refer the following blog - Tech Blogand let us know if the issue persists
Sai Prasanna Sinde 4,260 Reputation points Microsoft External Staff

2025-02-18T01:49:20.06+00:00

Hi @Morgan Ecklund

Just checking in to see if you had a chance to see my response to your question. Please tell us if it was helpful and feel free to reach out to us if you have any queries.
Alex Burlachenko 1,750 Reputation points

2025-02-19T13:21:11.64+00:00

Hey Morgan,

Ah, the dreaded 502 Azure’s way of saying, "I see your traffic, but I’m just gonna yeet it into the void." Sounds like your App Gateway and App Service are having a proper domestic over private endpoints and VNet integration.

First, check if your App Service is actually listening on the private IP (it’s like checking if your mate’s actually home before knocking). Make sure the private endpoint is properly linked and the DNS is resolving to the private IP, not the public one. Also, double-check your NSGs and route tables no one likes a blocked gatecrasher.

If all else fails, try turning off and on again (the IT version of "have you tried talking it out?"). And if it’s still being a diva, raise a ticket with Azure support they’re the couples counsellors for this kind of drama.

Let me know if you sort it. Cheers!

Alex
Sarthak Agarwal 81 Reputation points Microsoft Employee

2025-02-21T06:08:49.64+00:00

Agree with KapilAnanth-MSFT's responses. Please initiate a Support ticket for live troubleshooting session from a Microsoft Expert.

Thanks,
Sarthak.

1 answer

Sai Prasanna Sinde 4,260 Reputation points Microsoft External Staff

2025-02-17T05:04:38.58+00:00

Hi @Morgan Ecklund

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

In addition to the solution provided above by @Vidya Viraktamath , I wanted to add a few more details.

Make sure the NSG associated with the Application Gateway subnet allows outbound traffic to the private IP address of your Application Service on the port it's listening on (443 for HTTPS) and also verify the NSG on the Application Service subnet allows inbound traffic from the Application Gateway subnet on the service port. Please refer this document.

If you have a UDR on the Application Gateway subnet, make sure it's not inadvertently routing traffic destined for the Application Service to an incorrect location. The route for the Application Service's private IP range should point directly to the virtual network or have a more specific route. Please refer this document.

The Application Gateway needs to resolve the private FQDN of your Application Service to its private IP address. Make sure that the Application Gateway is using a DNS server that can resolve private DNS records within your virtual network.

The presence of a custom DNS in the VNet could also cause issues. An FQDN used for backend pool members might not resolve correctly by the user configured DNS server for the VNet.

Check that the health probes configured on your Application Gateway are correctly set up and are able to reach your Application Service. If the probes are failing, the Application Gateway will mark the backend as unhealthy and return 502 errors.

Make sure the Private Endpoint for your Application Service is correctly configured and associated with the correct subnet and also verify that the private DNS zone for your Application Service has the necessary A records that map the service's FQDN to the private IP address of the Private Endpoint.

Also use the "Effective Security Rules" feature in the Azure portal to check the actual NSG rules that are applied to the Application Gateway and Application Service subnets.

Enable diagnostic logging for your Application Gateway and analyze the logs for any errors or clues related to the 502 error. Check the logs of your Application Service for any errors or exceptions.

References: Troubleshooting bad gateway errors in Application Gateway

Please refer to this document for additional reference and see if it works for you.

Kindly let us know if the above helps or you need further assistance on this issue.
Please sign in to rate this answer.
Morgan Ecklund 0 Reputation points

2025-02-18T21:34:16.9633333+00:00

@Sai Prasanna Sinde and @Vidya Viraktamath , Thank you for your responses.

We currently have an App gateway front end with a Web Application firewall serving up multiple Web app service Backends. We shall call them Forms and Web. See Figure 2 could not attach will share later (if I can).

The Health probes are in place and working fine.

The goal of the project is to Isolate the backends further by putting them on private networks (Subnets) and a Firewall for enhanced feature such as IDPS and Threat Intelligence. between the App gateway Vnet and the Web app service backends Vnet. Here is what what we are going for See attached Graphic (I could not attach graphics) There are no Network security groups assigned to the Vnets.

I would love to be able to confirm DNS is resolving correctly, but can not from the the App gateway.

I am implementing Private networking and a firewall together. (created a network rule to allow all traffic from the App) I created the UDRs for the app gateway to the firewall and from the.

Venkat V 775 Reputation points Microsoft External Staff

2025-02-19T12:01:16.0233333+00:00

Hi @Morgan Ecklund

You can try using a hostname override in your backend HTTP settings with 'hostname' as shown below and check if it works.

Or use below option to pick from backend target

Since you have already mentioned that the backend health is healthy but are still receiving 502 errors, you may check the following steps.

I suggest you start troubleshooting the issue by following this guide: Troubleshooting bad gateway errors in Application Gateway

The certificate on the listener requires the entire certificate chain to be uploaded, including the root certificate from the CA, intermediates, and the leaf certificate, to establish the chain of trust. For more details, refer to Upload the root certificate to Application Gateway's HTTP Settings.

You can refer to the link below for steps on how to check an incorrectly bundled certificate and how to fix it

https://learn.microsoft.com/en-us/answers/questions/51336/appgateway-v2-certificate-issue

Alternativly, You can use service endpoint as a simple solution If the subnet contains service endpoints, the application gateway will route the request to the app service via its private IP address. DNS resolution is based on a private DNS zone or custom DNS server, if configured, or it uses the default Azure-provided DNS.

Reference: - Integration with App Service via service endpoint
Integration with App Service via private endpoint

I hope this helps to resolve your issue.

If the above is unclear and/or you are unsure about something, add a comment below.

KapilAnanth-MSFT 48,791 Reputation points Microsoft Employee

2025-02-19T13:53:33.67+00:00

@Morgan Ecklund ,

Greetings.

I see Vallepu Venkateswarlu and Sai Prasanna Sinde has provided some of their analysis.

Point to note :

If you are going to place an App service behind an App Gateway, and want the traffic to be private only - you should use Private EndPoint only

VNet Integration has no scope in this

Analysis,

Unfortunately, we cannot check DNS Resolution from an App Gateway instance.

What can be done is

You should deploy a VM in the same VNet as the App Gateway

And check the DNS resolution from this VM

This should you give you an overview of whether or not DNS works.

Anyhow, since Health Probes are succeeding, I believe DNS is resolving correctly.

Next Steps,

Did you check this set up without using Azure Firewall?

Did that work?

Or were you not able to test this out before deploying the Firewall directly?

In this doc : Firewall and Application Gateway for virtual networks, can you specify which set up are you using?

Is it Firewall and Application Gateway in parallel

or Application Gateway before Firewall

or Application Gateway after firewall

I'd highly appreciate if you could share an end to end architecture diagram

For 502, did you check the Access log ?

The fields "httpStatus" and "serverStatus" should tell us the HTTP status returned by App Gw and the backend respectively.

Also, "serverRouted" field should tell you if the DNS resolution was successful or not

Can you run the steps mentioned in Methods to view Backend health?

Azure Network Watcher's Connection Troubleshoot

and Backend server certificate visualization

Cheers,

Kapil

Morgan Ecklund 0 Reputation points

2025-02-20T14:45:17.0033333+00:00

@KapilAnanth-MSFT Thank You!

That really Tied it all together.

For the next steps:

the document you refer to (Firewall and Application Gateway for virtual networks) is what I am using to design the environment. I have a document I can share share. Not sure how to get it to you.
I did as you advised and setup a VM on the same Vnet. The VM can not resolve the FQDN for the app Service Back end!!!
SO I think that is the first thing I need to overcome.
the Next thing is the UDR.
I had an issue with this when I was setting up my Dev/test environment, which is now working.
I contacted MS support and they said a /32 route was need to route the traffic correctly.
Is this true? Right now my route is /26 (to route all the traffic for the subnet through the firewall).
The App Services that are behind the app gateway need to communicate with SQL servers that are also on the private Subnets networks right now. The support engineer also said I had to use the Virtual Network Integration to connect to those resources in the other private networks.

Is this true?
I am doing this work over the week end as this is a production system. So the access logs will not be that helpful right now. I will plan on running the methods to determine backend health when I am working on this over the weekend.

Thanks again for the Support
Regards

Morgan

KapilAnanth-MSFT 48,791 Reputation points Microsoft Employee

2025-02-20T16:44:45.2833333+00:00

@Morgan Ecklund , You are most welcome.

I take it that you have an existing support ticket open with Microsoft. I am initiating a private conversation, please share the SR Number.

Wrt, "I did as you advised and setup a VM on the same Vnet. The VM can not resolve the FQDN for the app Service Back end!!!"

Then you should first make sure the DNS resolution is working as expected.

For this, make sure the Private DNS Zone is properly linked to the VNET of the App Gateway

If you are using a custom DNS Server in the VNET, make sure it correctly resolves the App Service's FQDN to the Private EndPoint's IP

See : Azure Private Endpoint DNS integration

Wrt, "Firewall and Application Gateway for virtual networks"

Can you please specify which out of the three I mentioned you are trying to employ.

Only based on this, I would be able to comment on the UDRs involved/required

Wrt, "I contacted MS support and they said a /32 route was need to route the traffic correctly. Is this true?"

This could be the case, especially with Private EndPoints.

Note that Private EndPoint introduces a /32 Default Route in the VM's route tables (VMs connected to VNET where the Private EndPoint is created)

To override this with Route Table, you must create a /32 custom route User Defined Route

Wrt, "The App Services that are behind the app gateway need to communicate with SQL servers that are also on the private Subnets networks right now. The support engineer also said I had to use the Virtual Network Integration to connect to those resources in the other private networks.

Is this true?"

Correct.

This is the general idea

If you want to initiate private traffic from a VM/resource in VNET to a PaaS Service - use Private EndPoint for the PaaS Service.

If you want to initiate private traffic from a PaaS Service to resources in a VNET (VMs and PEs) - use VNET integration for the PaaS Service.

In your case, it's a combination of both

For App Service to speak to SQL Database via private network

App Service should be integrated to a VNet

SQL DB should be having a private endpoint in the same VNET (or Peered VNET)

So the net effective traffic path becomes, (without Firewall)

App Gateway ----> (App Service via PE ---- App Service via VNET Integration) ----> SQL via PE

Note that the ( ) indicates traffic is contained within the App Service

i.e.,

AppGw(Source) to App Service PE(Destination)

App Services VNET integration(Source) to SQL PE(Destination)

I suggest you work with the Support Engineer, as they have better access to the internal logs and architecture.

Cheers,

Kapil
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Cannot connect to Application Service from the Application Gateway when Private endpoints and Virtual Network Integration

1 answer

Your answer