Client not receiveing response from Application Gateway timing out and collecting 400 httpStatus

Matteo Mario Cossu 21

I have a NodeJS Express API.
It exposes an endpoint expecting a multipart form to collect files.
It is deployed in an AKS instance.
The AKS is behind an Application Gateway.

A client is performing several serial calls toward the endpoint described above.
At some point, at POST request N of M, where N < M, it stops receiving an answer, simply timing out.
There's no pattern in the failing requests.
The application gateway logs (AzureDiagnostics table) display a record with dimension 'httpStatus_d' valued 400 and no 'serverStatus_s' dimension.
The API logs display no request, meaning the request is not being forwarded by the application gateway.

I removed the WAF_v2 and redeployed the AGW as a Standard_v2 tier (no waf, so no policies should block the request at this point) and still experiencing the issue.

I need to:

understand what's causing the timeout / 400 issue
find a solution for the issue

MMC

KapilAnanth-MSFT 46,956 Reputation points Microsoft Employee

2024-10-21T13:53:20.9433333+00:00

@Matteo Mario Cossu , Thanks for keeping us updated.

Sure, let us know how it goes.

(I have redacted the SR# as this is considered PII)

Cheers,

Kapil

2 answers

Sina Salam 12,011

Hello

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Regarding your explanation:

You can adjust the maxRequestBodySize on your Application Gateway to handle larger file uploads using Azure CLI, for example - https://learn.microsoft.com/en-us/azure/application-gateway/configuration-overview#http-settings:

  az network application-gateway http-settings update \
      --gateway-name myAppGateway \
      --resource-group myResourceGroup \
      --name appGatewayBackendHttpSettings \
      --max-request-body-size 10485760 # For 10MB

Also, make sure that both the client request timeout and the backend timeout settings on the AGW are long enough to accommodate the upload, the example shows Azure CLI bash command:

    az network application-gateway http-settings update \
      --gateway-name myAppGateway \
      --resource-group myResourceGroup \
      --name appGatewayBackendHttpSettings \
      --timeout 180 # Increase to 3 minutes, or adjust as needed

Other things you can do is to check Logs for Patterns, Check NodeJS API Scaling, and check Client-Side Request Behavior to handle multiple and serial request. https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502?source=recommendations

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Matteo Mario Cossu 21 Reputation points

2024-10-15T14:59:46.8366667+00:00
Hi @Sina Salam

Thanks for the answer. Yet:

maxRequestBodySize should not be a problem.
The client is making a single serial call for each file that has a size <= 1MB

timeout should already be at 300s

I'm not sure what Patterns are you referring to.

The NodeJS scaling should not be a problem, the request does not reach the NodeJS API.

Client-Side Request Behavior: the client is executing single calls, in serial, for each file, with a 2s sleep time between each call (as an experiment)
Sina Salam 12,011 Reputation points

2024-10-15T19:50:57.2066667+00:00
Hello Matteo Mario Cossu

Thank you for your feedback.

I will advise you to investigate request limits, even though you're dealing with relatively small files (<=1MB), Application Gateway has some built-in limitations regarding request handling. Perhaps, you've already set the maxRequestBodySize appropriately, but it could be worth reviewing other request-related limits, like:

Header Size Limits: Large headers in the request could be causing issues.

Connection Limits: AGW has connection limits that could come into play if you're making multiple requests quickly, even if they are serial.

https://learn.microsoft.com/en-us/azure/azure-subscription-service-limits#application-gateway-limits

Also, increase the sleep time a bit, reviewing detailed logs, ensuring health probes and load balancing are correctly configured, and checking network latency between AGW and AKS, you may be able to pinpoint the root cause of the timeout and 400 errors.

Cheers

Matteo Mario Cossu 21

Hello @Sina Salam
from the application-gateway-limits article:

Resource	Limit	-
Maximum file upload size (Standard SKU)	V1 - 2 GB; V2 - 4 GB;	We're nowhere near that ( < 1MB)
Maximum header size	32 KB	Manually checked, we're nowhere near that size per single header or even for the entire request headers block (< 1KB)
Maximum URL length	32 KB	We're definitely below 1 KB
Concurrent WebSocket connections	20K	20k per instance. We might have 1 WS, top, at the moment...

I can't find connection limitations in the article, anyway we're a t 1 or 2 connection at once (it's a test environment with 2 users).
I have no WAF enabled, so no policies, hence no custom limitations for maxRequestBodySize, meaning it must be up to 4 GB
Sleep time between calls is already high (2000ms).
Anyway it's a gateway, it should handle multiple connections at once, i can't believe AGW might have problems with 2 close upload calls of small size
I have no other logs at my disposal but those provided by the diagnostics settings, and AzureMetrics table provides no useful insights
I Can't measure network latency between AGW and AKS, but an average 200 request takes about 700ms for a complete roundtrip (timeTaken_d logs field)
Load balancing and Health probes to default settings, all instances are healthy

Sina Salam 12,011 Reputation points

2024-10-17T10:44:32.46+00:00
Hello Matteo Mario Cossu,

Thank you for your feedback.

Given the randomness of the failures and the fact that requests are not reaching the backend, it’s likely an intermittent networking or AGW configuration issue. This is my final checklists for this issue:

Since the requests are not reaching the NodeJS API, it’s crucial to check whether the AKS backend pool is intermittently failing to communicate with the Application Gateway. Try performing the following steps: Run network connectivity checks between the Application Gateway and the AKS cluster using tools like curl or ping from a pod in AKS to ensure there are no transient network issues. Make sure the health probes for the backend pool are properly configured to accurately reflect the status of the AKS instances. If health probes are misconfigured, AGW may mark instances as unhealthy and stop forwarding requests to them, leading to timeouts.

The absence of detailed serverStatus_s logs (showing backend server response codes) suggests that AGW isn’t forwarding the requests. I will advise you again please investigate the access and request tracing logs:

Yes, even though the setup is for testing with few users, connection limits and scaling on the backend could be contributing to the issue. Please, make sure your backend pods in AKS are properly scaled to handle multiple serial requests efficiently, such as using horizontal pod autoscaling to handle variations in request load. It’s possible that some requests fail due to a lack of available backend resources at the moment AGW forwards the request. Also, AGW may be hitting a limit on the number of backend connections. You will need to explicitly set connection limits higher on the backend HTTP settings using the following:
az network application-gateway http-settings update \ --gateway-name myAppGateway \ --resource-group myResourceGroup \ --name appGatewayBackendHttpSettings \ --connection-draining true --drain-timeout 300 \ --max-connections 1000 # Increase the limit as needed

Now at this point, consider adjusting the timeout on the AGW further and explicitly setting connection draining parameters, which help AGW gracefully manage connections when backend resources are under load:
az network application-gateway http-settings update \ --gateway-name myAppGateway \ --resource-group myResourceGroup \ --name appGatewayBackendHttpSettings \ --timeout 600 # Increase as needed

Make sure there are no networking issues in the AKS setup, such as bottlenecks or firewall rules blocking requests from AGW at random intervals and if AGW and AKS are in different virtual networks, ensure that VNet peering is correctly set up with proper DNS resolution between the two.

Last but not the least, since the client makes multiple serial requests, ensure that keep-alive connections are enabled between the client and the Application Gateway and between AGW and AKS to avoid frequent connection setups and teardowns: Your NodeJS Express API should support persistent connections and adjust keep-alive settings in the AGW using the following:
az network application-gateway http-settings update \ --gateway-name myAppGateway \ --resource-group myResourceGroup \ --name appGatewayBackendHttpSettings \ --keep-alive true

Therefore, if the above could not solve your issues, with what I and @KapilAnanth-MSFT have suggested, it means there is an overlook or something you've done somewhere that are yet to be known. If you get it resolve, we would be very glad to read and learn from you. Thank you.

KapilAnanth-MSFT 46,956 Microsoft Employee

@Matteo Mario Cossu ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I see you have checked the Access log.

For the field "serverStatus"
May I ask what is the value in the field "serverRouted"
This will confirm if the App gateway did forward the request to backend or not.

Also,

Note that "400 – Bad Request" is sent as a response when a request isn't compliant to RFC.
See : 4XX response codes (client error)
- You can see a huge list of reasons as to why 400 response is received.
Can you please confirm if the "N" number of requests are actually proper HTTP/HTTPS request and there is no conflict with the above mentioned reasons?

Do you perform file upload operations?

Cheers,

Kapil

Matteo Mario Cossu 21

Hello @KapilAnanth-MSFT

i checked the 'serverRouted' field, it's blank on the record of the 400 httpStatus.

TimeGenerated	httpMethod_s	clientIP_s	requestUri_s	httpStatus_d	serverStatus_s	timeTaken_d	serverRouted_s
10/14/2024, 10:36:52.000 AM	POST	5X.XXX.XXX.X2	/scraper-api/blob/loose	400	59.937
10/14/2024, 10:35:52.000 AM	POST	5X.XXX.XXX.X2	/scraper-api/blob/loose	200	200	0.626	192.168.80.55:5000
10/14/2024, 10:35:51.000 AM	POST	5X.XXX.XXX.X2	/scraper-api/blob/loose	200	200	0.656	192.168.80.55:5000
10/14/2024, 10:35:50.000 AM	POST	5X.XXX.XXX.X2	/scraper-api/blob/loose	200	200	0.625	192.168.80.55:5000

I already verified the list from the 400--bad-request list and it seems like we are not in any the indicated scenarios.I can confirm the requests are coherent between each other.The post calls are intended to perform file upload operations via the node.js blob storage SDK.
The client executing the call is a .NET module using the following:

System.Net.ServicePointManager.ServerCertificateValidationCallback = _
  Function(se As Object, _
  cert As System.Security.Cryptography.X509Certificates.X509Certificate, _
  chain As System.Security.Cryptography.X509Certificates.X509Chain, _
  sslerror As System.Net.Security.SslPolicyErrors) True


Dim client As System.Net.Http.HttpClient = New System.Net.Http.HttpClient()
client.Timeout = TimeSpan.FromMinutes(1)
client.DefaultRequestHeaders.Add("authorization", "Bearer " + in_token)
Dim form As System.Net.Http.MultipartFormDataContent = New System.Net.Http.MultipartFormDataContent()
form.Add(New System.Net.Http.StringContent(in_id), "id")
form.Add(New System.Net.Http.StringContent(in_source), "source")
Dim file1 As System.Net.Http.ByteArrayContent = New System.Net.Http.ByteArrayContent(File.ReadAllBytes(in_filePath))
file1.Headers.ContentType = New System.Net.Http.Headers.MediaTypeHeaderValue(in_contentType)
form.Add(file1, "loose", (New FileInfo(in_filePath).Name))

Dim response As System.Net.Http.HttpResponseMessage = client.PostAsync(in_url+"/blob/"+in_blobType, form).Result
out_response = response.Content.ReadAsStringAsync.Result
out_statusCode = response.StatusCode.AsText

Matteo Mario Cossu 21 Reputation points

2024-10-15T14:47:34.08+00:00
Hi @KapilAnanth-MSFT

the serverRouted for the record in 400 is blank

we are not in any of the scenarios for the 4XX errors (already checked)

the N requests are proper requests with no conflict against the RFC compliance rules

Yes, we perform upload operations
KapilAnanth-MSFT 46,956 Reputation points Microsoft Employee

2024-10-17T05:05:44.85+00:00
@Matteo Mario Cossu , Thanks for sharing the info.

In the same table, may I ask what is the value of the field "error_info" ?

With the value in the field, can you please cross check 4XX Errors Table (right below the Access Log table)

Cheers,

Kapil
Matteo Mario Cossu 21 Reputation points

2024-10-17T10:08:06.7133333+00:00

Hello @KapilAnanth-MSFT
the error info code for that request is 'ERRORINFO_NO_ERROR'.
So no useful insight, there.
KapilAnanth-MSFT 46,956 Reputation points Microsoft Employee

2024-10-21T12:52:58.2766667+00:00

@Matteo Mario Cossu ,

Strange, especially when WAF is not used/disabled. (Can you confirm you are not using WAF while you captured the above log once again, please?)

To troubleshoot further, we will need a specialized 1:1 session, where a support engineer can check the backend logs to pinpoint the issue. If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

P.S : We will be able to provide you one-time support if and only if your subscription is not managed by a CSP

Cheers,

Kapil
Matteo Mario Cossu 21 Reputation points

2024-10-21T13:46:55.03+00:00

i can confirm the WAF is disabled.
I replicated the issue in a subscription with support plan, and I opened a ticket (TrackingID#<REDACTED>).
I've been receiving support and a first attempt at an answer related to some Application Gateway 'path rule' definition (the path rule resources are not directly handled by our deploy scripts, so it must be due to some kind of Terraform / AKS apply automatism).
I'll add a more specific answer in this thread as soon as something comes up.
KapilAnanth-MSFT 46,956 Reputation points Microsoft Employee

2024-10-21T13:52:09.1733333+00:00

@Matteo Mario Cossu , Thanks for keeping us updated.

Sure, let us know how it goes.

(I have redacted the SR# as this is considered PII)

Cheers,

Kapil

Share via

Client not receiveing response from Application Gateway timing out and collecting 400 httpStatus

2 answers

Your answer