ACS60021: The request has been terminated by the server (tenant exceeded rate limit)

In case you are using ACS aggressively sometime you may hit upon the following error. 

 ACS60021: The request has been terminated by the server (tenant exceeded rate limit). Trace ID: d80e94a4-8571-45cc-82f1-b900f7f3ce16. Timestamp: 2013-07-26 01:39:25Z

 ----- Exception -----

 Expected:

 Actual: System.Data.Services.Client.DataServiceQueryException: An error occurred while processing this request. --->
System.Data.Services.Client.DataServiceClientException: <?xml version="1.0" encoding="utf-8" standalone="yes"?>

 <error xmlns="https://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
 <code></code>
 <message xml:lang="en-US">ACS60021: The request has been terminated by the server (tenant exceeded rate limit). Trace ID: d80e94a4-8571-45cc-82f1-b900f7f3ce16.
Timestamp: 2013-07-26 01:39:25Z</message>
 </error>
 at System.Data.Services.Client.QueryResult.ExecuteQuery()
 at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, QueryComponents queryComponents)
 --- End of inner exception stack trace ---
 at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, QueryComponents queryComponents)
 at System.Data.Services.Client.DataServiceQuery`1.Execute()
 at System.Data.Services.Client.DataServiceQuery`1.GetEnumerator()
 at Microsoft.DataTransfer.Authentication.AcsToken.Management.AcsManagementServiceExtensions.DeleteServiceIdentityIfExists(ManagementService svc, String name) in

You might have hit a cul-de-sac while trying to find more information considering that the error code and error message didn’t result in any relevant information by search engines. So I dug in deep and got the real issue behind it. But not anymore and it's time to break the mystry.

This is due to the reason that ACS rejects token requests when ACS internal resources are temporarily consumed by a high token request rate from all namespaces. In this case, ACS returns an HTTP 503 "Service unavailable" error with the ACS90046 or ACS60021 (ACS Busy) error codes.

We have now documented the limitations of ACS service at https://msdn.microsoft.com/en-us/library/gg185909.aspx. The page with error codes https://msdn.microsoft.com/en-us/library/gg185949.aspx has also been updated for ACS90046 (ACS busy) or ACS60021.

Please refer this page with ACS service limitations and the page with details on ACS retries guidelines, https://msdn.microsoft.com/en-us/library/jj878112.aspx.

Hope this blog gave you an insight into the issue and cleared a few threads of doubt around ACS.