Application Gateway Issue

Mohammed Shafi 20

Hello Team,

We are experiencing an issue with our application running in Azure VM and accessing through the application gateway.

Some functions are running after page refresh, but those are not working properly after actions like submit or cancel, etc. on the application side.

If we run the application directly without a gateway setup it is working fine. But, if it is running through the gateway even if I disable the WAF these backend events are not working.

We are using the ERPnext application deployed in Azure VM, accessing through Azure Application Gateway Tier WAF v2.

If someone can provide some clarity or guidance on how to resolve this issue it would be greatly appreciated.

Thanks in advance

Mohammed Shafi 20 Reputation points

2025-01-08T16:12:19.78+00:00

I disabled the WAF completely, changed the Application Gateway Tier to Basic mode, and disabled HTTP2 requests, but the problem continues.
Sai Prasanna Sinde 3,585 Reputation points Microsoft Vendor

2025-01-08T21:12:35.7733333+00:00
Hi @Mohammed Shafi

Welcome to the Microsoft Q&A Platform. Thank you for posting your query here.

Please go through the below steps:

Application Gateways can disrupt session state if not configured correctly. After an action like submit or cancel, the subsequent request might be routed to a different server or lose necessary session information, leading to the issue. ERPNext, like many applications, likely relies on cookies to maintain session state.

In your Application Gateway, go to your backend pool settings and enable Cookie-based affinity.

This ensures that requests from a specific client are always routed to the same backend server within the pool.

If your application uses a specific session cookie name, you might want to specify it in the Application Gateway settings for more precise handling.

If you're unsure, leaving it blank is generally fine as the Application Gateway will generate a cookie name.

Make sure ERPNext is configured to use cookies for session management and that the cookie settings are appropriate for your setup. For your reference: https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/session-affinity?tabs=session-affinity-gateway-api#how-to-configure-session-affinity

Application Gateway might be modifying headers that ERPNext needs to function correctly.

Use your browser's developer tools (Network tab) to capture the request headers when accessing the application directly (working) and through the Application Gateway (not working).

Carefully compare these headers, Any headers related to sessions, authentication or CSRF protection.

If you identify any missing or modified headers that ERPNext requires, you can potentially add them back using Rewrite rules in the Application Gateway. This is a more advanced configuration. For your reference: https://learn.microsoft.com/en-us/azure/application-gateway/tutorial-http-header-rewrite-powershell#specify-your-http-header-rewrite-rule-configuration

Long-running operations in ERPNext might be timing out at the Application Gateway level.

In your Application Gateway's HTTP settings, adjust the Request timeout value to a higher value, like 120 or 180 seconds, to see if this helps.

You can increase it further if needed, but very long timeouts can sometimes indicate other underlying problems. For your reference: https://docs.azure.cn/en-us/application-gateway/application-gateway-faq#what-are-the-settings-for-keep-alive-timeout-and-tcp-idle-timeout

Even though you said direct access is working, there might be intermittent issues with your backend VM or ERPNext itself that only manifest under the load or specific conditions imposed by the Application Gateway.

Use the Backend health feature in Application Gateway to monitor the health status of your VM. Look for any errors or warnings. For your reference: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-backend-health#methods-to-view-backend-health

Examine the ERPNext logs on your VM for any errors or exceptions that might be occurring around the time of the failures.

Check Azure VM diagnostics for resource utilization like CPU, memory, network to make sure the VM isn't becoming overloaded. For your reference: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/performance-diagnostics https://learn.microsoft.com/en-us/azure/virtual-machines/boot-diagnostics

Problems with SSL/TLS certificates or configurations can cause unexpected behavior.

Make sure the SSL/TLS certificate you're using on the Application Gateway is valid and trusted.

If you're using end-to-end SSL, ensure the certificate on your backend VM is also valid and properly configured. For your reference: https://learn.microsoft.com/en-us/azure/application-gateway/ssl-overview

If any caching is configured at the Application Gateway level, it could be interfering.

If you have caching enabled on the Application Gateway, try disabling it temporarily to see if it's causing the issue.

After making each configuration change, thoroughly test your application to see if the issue is resolved.

Kindly let us know if the above helps or you need further assistance on this issue.

Thanks,

Sai.
Mohammed Shafi 20 Reputation points

2025-01-09T10:28:52.7333333+00:00

Thank you so much for your detailed explanation regarding our queries.

As you mentioned, I enabled the Cookie-based affinity and Request timeout value to a higher value, like 120 and confirmed other related settings.

However, I noticed the difference in the application request's response header running behind the Application Gateway. The Connection == "keep-alive" is missing, but it is present in every GET/POST request without gateway setup.

Does this setting cause this issue? What are your thoughts on this? I tried to rewrite the header using the Application Gateway re-write method to include it in the header, but it did not allow me to write Connection == "keep-alive" and I am getting errors like Microsoft doesn't support this feature currently.

Please let me know your suggestion.

Thanks.
Sai Prasanna Sinde 3,585 Reputation points Microsoft Vendor

2025-01-09T21:28:55.4766667+00:00
Hi @Mohammed Shafi

Good day!

Please go through the below points:

The absence of Connection: keep-alive in the response headers observed behind the Application Gateway suggests that the connection between the Application Gateway and your backend VM might not be persistent.

This means for every action you perform in the application, a new connection is being established. This can lead to Lost Session State, Performance Degradation and Intermittent Failures.

Application Gateway V2 supports HTTP/2, which handles connections more efficiently than HTTP/1.1. If you haven't already, try enabling it on the Application Gateway listener that handles traffic to your ERPNext application. You have mentioned that you have already tried disabling HTTP/2. Kindly enable it.

HTTP/2 should generally improve performance and handle multiple requests/responses over a single connection better than HTTP/1.1. Please go to your application gateway's configuration, you can find it under settings and enable HTTP/2. For your reference: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-faq#how-do-application-gateway-and-azure-load-balancer-differ

In your Application Gateway, go to backend settings and check the HTTP settings associated with your ERPNext backend pool.

Make sure the backend protocol is set to HTTP or HTTPS if you're using end-to-end SSL. The key is to ensure that this connection defaults to keep-alive and if it is set to HTTPS, change it to HTTP and check.

Make sure that your web server on the Azure VM is configured to support persistent connections.

The Connection header is considered a hop-by-hop header, which means it's relevant to the immediate connection between two parties. Application Gateway, like many proxies, intentionally restricts modification of these headers to prevent unexpected behavior.

We think trying to force Connection: keep-alive through a rewrite is generally not a proper way. The goal is to ensure that both the Application Gateway and your backend server are configured to use persistent connections by default for HTTP/1.1 traffic.

Kindly let us know if the above helps or you need further assistance on this issue.

Thanks,

Sai.
Sai Prasanna Sinde 3,585 Reputation points Microsoft Vendor

2025-01-10T18:48:17.2+00:00

Hi @Mohammed Shafi

Good day!
Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

Thanks,

Sai.
Sai Prasanna Sinde 3,585 Reputation points Microsoft Vendor

2025-01-13T18:58:59.4233333+00:00

Hi @Mohammed Shafi

Greetings!
Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

Thanks,

Sai.
Mohammed Shafi 20 Reputation points

2025-01-14T14:34:59.9533333+00:00

Thanks, Sai Prasanna Sinde for your follow-up.

We overcome this issue by manually reloading the page (location.reload()) after the post-event of application action buttons.
Sai Prasanna Sinde 3,585 Reputation points Microsoft Vendor

2025-01-15T16:55:02.0166667+00:00

Hi @Mohammed Shafi

I'm glad that you were able to resolve your issue.Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others", I'll repost your solution in case you'd like to "Accept" the answer so that other community members facing the same issues can find the right answers easily.

Thank you again for your time and patience throughout this issue.

Cheers,

Sai.

Accepted answer

Sai Prasanna Sinde 3,585 Reputation points Microsoft Vendor

2025-01-15T16:58:09.9266667+00:00

Hi @Mohammed Shafi

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:
Some functions are running after page refresh, but those are not working properly after actions like submit or cancel, etc. on the application side.

If we run the application directly without a gateway setup it is working fine. But if it is running through the gateway even if I disable the WAF these backend events are not working.

Solution:
You have overcome this issue by manually reloading the page after the post-event of application action buttons.

Thanks,
Sai.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Application Gateway Issue

0 additional answers

Your answer