Azure Front Door resource response with a lot of 429 HTTP errors

Tomasz Sieńko 0

Hi,

We manged to deploy a Azure Front Door resource in front of a website.

In reports and metrics we see aprox 100 000 - 200 000 request/h, so I assume we don't fall into 100 000 req/s limit. Still, sometimes our developers our clients get 429 HTTP errors ( too many requests). It's critical, cause of important assets that couldn't load in some cases.

I've tried to dig into logs in LogAnalytics and I see 429 errors, but without any interesting informations. I would like to add, that before we deployed Front Door, there were never 429 http problems, so it's defenitely something with POP location or FD itself. WAF is completely turned off. We also want to try with cache, and custom rules for caching, but we are clueless why it is happening.

Did anyone enounterd same issue ? Are there any solutions ?

KapilAnanth-MSFT 44,556 Reputation points Microsoft Employee

2024-09-13T06:48:48.41+00:00
@Tomasz Sieńko ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are experiencing HTTP 429 responses in your service.

Can you confirm if the 429 is from the AFD and not from the backend?

If you look into the Access log, you can see if the response is from AFD or the Origin

Can you share one such instance of Access log where the HTTP Status is 429

Cheers,

Kapil
Tomasz Sieńko 0 Reputation points

2024-09-16T08:51:48.4833333+00:00

Hi @KapilAnanth-MSFT ,

I've checked logs on the backend server and 429 error are nowhere to be found.
I only saw those errors on the access.log in Azure Front Door, below is an example:

trackingReference_s sni_s requestBytes_s responseBytes_s clientIp_s clientPort_s socketIp_s timeToFirstByte_s timeTaken_s requestProtocol_s securityProtocol_s rulesEngineMatchNames_s httpStatusCode_s httpStatusDetails_s pop_s cacheStatus_s errorInfo_s ErrorInfo_s

20240905T084129Z-1658564db57mq75202120ra2tw0000000cy0000000011c5n mysite.com 223 1418 37.109.0.0 6486 37.109.0.0 0 0 HTTPS TLS 1.3 [] 429 429 WAW CONFIG_NOCACHE UnspecifiedClientError UnspecifiedClientError
KapilAnanth-MSFT 44,556 Reputation points Microsoft Employee

2024-09-17T07:48:37.7733333+00:00
@Tomasz Sieńko ,

Thanks for sharing the log.

From this, "httpStatusDetails_s" indicates the response from the Origin

As we can see, the Origin is the one responding with 429

AFD is simply forwarding the same, it is not giving 429 by itself.

As next steps,

I would suggest you consider checking the Origin application's logs to understand why there is 429 HTTP Responses.

Also, it is possible that AFD Health probes are sending a huge number of requests to your Origin, so making the Origin respond with a 429

If you have only one Origin, consider Disabling health probes

If not, try to reduce the probe volume as mentioned in AFD Health probes

You can do this by using HEAD method for health probes or increase the Probe interval to a higher value (lesser frequency)

Since each Azure Front Door edge location is sending health probes to your origins, the health probe volume for your origins can be quite high. The number of probes depends on your customer's traffic location and your health probe frequency. If the Azure Front Door edge locations doesn’t receive real traffic from your end users, the frequency of the health probe from the edge location is decreased from the configured frequency. If there is traffic to all the Azure Front Door edge locations, the health probe volume can be high depending on your health probes frequency. An example to roughly estimate the health probe volume per minute to an origin when using the default probe frequency of 30 seconds. The probe volume on each of your origin is equal to the number of edge locations times two requests per minute. The probing requests will be less if there is no traffic sent to all of the edge location

In essence,

Your Origin should be capable of handling the health probe traffic + actual traffic

Your Origin should be capable of handing the expected traffic without giving a 429

Hope this helps

Cheers,

Kapil
Tomasz Sieńko 0 Reputation points

2024-09-17T09:07:53.6466667+00:00

@KapilAnanth-MSFT

I've checked the backend Nginx servers and "429" HTTP error is nowhere to be found, there are no limitations and no rating configured on the origin side. There were no errors before configuring FD.

Also AFD Health Probe for origin group was disabled when creating the resource.
KapilAnanth-MSFT 44,556 Reputation points Microsoft Employee

2024-09-18T05:40:56.19+00:00

@Tomasz Sieńko ,

This is strange, provided health probes are disabled and there is no WAF Enabled on the AFD.

In this case, to troubleshoot further, we will need a specialized 1:1 session, where a support engineer can check the backend logs to pinpoint the issue.

If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

P.S : We will be able to provide you one-time support if and only if your subscription is not managed by a CSP

Cheers,

Kapil

Share via

Azure Front Door resource response with a lot of 429 HTTP errors

Your answer