How to fix and troubleshoot 503 errors in Azure Cosmos DB with substatus code of 20003

Katherine Bayona 0

Hi,

We are currently experiencing intermittent 503 errors in our cosmos db. Below is the error message:

ServiceUnavailable`` (503); Substatus: 20003; ActivityId: 0b3f26d8-f0ea-4d67-9260-c984144c7f7f; Reason: (GatewayStoreClient Request Timeout. Start Time UTC:01/20/2025 06:23:33; Total Duration:71507.1309 Ms; Request Timeout 65000 Ms; Http Client Timeout:65000 Ms; Activity id: 0b3f26d8-f0ea-4d67-9260-c984144c7f7f;)

Looking at the exception message it is showing "A task has been cancelled". When this happens, when we check the RU consumption, it is not at 100%.

What are the possible causes of this and what is the best way to troubleshoot?

Sai Raghunadh M 2,230 Reputation points Microsoft Vendor

2025-01-21T11:45:16.48+00:00

Hi @Katherine Bayona

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Sai Raghunadh M 2,230 Reputation points Microsoft Vendor

2025-01-20T11:37:06.4266667+00:00

Hi @Katherine Bayona

Thanks for the Question and using Microsoft Q&A.

It seems like you are encountering issues with Azure Cosmos DB. The 503 Service Unavailable error with substatus code 20003 signifies underlying I/O errors related to the operating system.

"A task has been cancelled" message and the fact that RU consumption is not at 100% indicate that the issue may be related to client-side timeouts or resource starvation.

Here are some steps to troubleshoot and potentially resolve this problem:

Please verify that your client machine is not experiencing high CPU or memory usage, as this can lead to timeouts.

https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-dotnet-sdk-request-timeout?tabs=cpu-new#high-cpu-utilization

Please ensure your application includes a robust retry mechanism to effectively handle transient errors. This will assist in recovering from temporary issues.

https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/conceptual-resilient-sdk-applications#should-my-application-retry-on-errors

consider adjusting the timeout settings for your client requests to allow additional time for operations to complete successfully. Please go through this similar thread for reference:

https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/63

Check for any network latency or connectivity issues between your client and Azure Cosmos DB

https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-dotnet-sdk#high-network-latency

Please go through this similar thread that might help you:https://learn.microsoft.com/en-us/answers/questions/1414611/microsoft-azure-cosmos-cosmosexception-response-st

Hope this helps. Do let us know if you any further queries. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
Katherine Bayona 0 Reputation points

2025-01-22T09:34:43.82+00:00

Hi, is there any recommendations or issues to help us narrow down the root cause for the below exception type captured from the diagnostics logs?

"ExceptionType":"System.Threading.Tasks.TaskCanceledException","ExceptionMessage":"A task was canceled."}

-We have ruled out the High CPU utilization as the values from the logs are not over 70% and the CPU measurements are taken within 10 seconds.
-We retries implemented on the application level.

-We are using singleton instance of the cosmos db client.

-We use Direct connection mode

Thanks

Sai Raghunadh M 2,230 Reputation points Microsoft Vendor

2025-01-22T10:57:25.0566667+00:00

Hi @Katherine Bayona

All the async operations in the SDK have an optional CancellationToken parameter. This CancellationToken parameter is used throughout the entire operation, across all network requests and retries. In between network requests, the cancellation token might be checked and an operation canceled if the related token is expired. The cancellation token should be used to define an approximate expected timeout on the operation scope.

Please go through this documentation that might help you:

https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-dotnet-sdk-request-timeout?tabs=cpu-new#cosmosoperationcanceledexception

I hope this information helps, please do let us know if you have any Queries.

Katherine Bayona 0 Reputation points

2025-01-22T11:29:44.6933333+00:00

Hi,

Should I be able to find this specific keyword: CosmosOperationCanceledException in the error message? As it is not showing at the moment, and I also cannot find the state of the CancellationToken through Cancellation Token has expired: true the diagnostics don't contain the context of the cancellation.

{"StartTimeUTC":"2025-01-22T02:19:23.7688320Z","DurationInMs":499.8788,"RequestUri":"[redacted]","ResourceType":"Document","HttpMethod":"POST","ActivityId":"98cb6ab1-0cfc-432d-ba2b-4d258360cc4a","ExceptionType":"System.Threading.Tasks.TaskCanceledException","ExceptionMessage":"A task was canceled."},

{"StartTimeUTC":"2025-01-22T02:19:24.2687879Z","DurationInMs":4999.8811,"RequestUri":"[redacted]","ResourceType":"Document","HttpMethod":"POST","ActivityId":"98cb6ab1-0cfc-432d-ba2b-4d258360cc4a","ExceptionType":"System.Threading.Tasks.TaskCanceledException","ExceptionMessage":"A task was canceled."},

{"StartTimeUTC":"2025-01-22T02:19:30.2679736Z","DurationInMs":65003.9257,"RequestUri":"[redacted]","ResourceType":"Document","HttpMethod":"POST","ActivityId":"98cb6ab1-0cfc-432d-ba2b-4d258360cc4a","ExceptionType":"System.Threading.Tasks.TaskCanceledException","ExceptionMessage":"A task was canceled."}]}}}]}]}]}]}]}]}]},

{"name":"Query Response Serialization","duration in milliseconds":0.0352}]}

In general, will switching to Gateway connection mode help?

Additional information is that it happens on ReadNextAsync calls.

Looking at the diagnostic logs below, we can rule out that it is not due to high cpu utilization. However, is there anything below, especially the number of open tcp connections, that indicate possible cause?

{"dateUtc":"2025-01-22T02:19:35.2763686Z","cpu":18.081,"memory":32126296.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1871,"availableThreads":32766,"minThreads":6,"maxThreads":32767},"numberOfOpenTcpConnection":23},

{"dateUtc":"2025-01-22T02:19:45.2766662Z","cpu":20.210,"memory":31993668.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0779,"availableThreads":32766,"minThreads":6,"maxThreads":32767},"numberOfOpenTcpConnection":23},

{"dateUtc":"2025-01-22T02:19:55.2769660Z","cpu":18.729,"memory":32043092.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0725,"availableThreads":32766,"minThreads":6,"maxThreads":32767},"numberOfOpenTcpConnection":23},

{"dateUtc":"2025-01-22T02:20:05.2774300Z","cpu":23.176,"memory":32136348.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1796,"availableThreads":32766,"minThreads":6,"maxThreads":32767},"numberOfOpenTcpConnection":23},

{"dateUtc":"2025-01-22T02:20:15.2777984Z","cpu":21.052,"memory":31935892.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1744,"availableThreads":32766,"minThreads":6,"maxThreads":32767},"numberOfOpenTcpConnection":23},

{"dateUtc":"2025-01-22T02:20:25.2780958Z","cpu":22.039,"memory":31920900.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0775,"availableThreads":32766,"minThreads":6,"maxThreads":32767},"numberOfOpenTcpConnection":23}]}},

Thanks

Sai Raghunadh M 2,230 Reputation points Microsoft Vendor

2025-01-22T12:04:48.5766667+00:00

Hi @Katherine Bayona,

In general, will switching to Gateway connection mode help?

Yes, Switching to Gateway connection mode might resolve your issue.

Please refer to this thread link: https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/63

Sai Raghunadh M 2,230 Reputation points Microsoft Vendor

2025-01-23T08:53:10.8033333+00:00

Hi @Katherine Bayona

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Sai Raghunadh M 2,230 Reputation points Microsoft Vendor

2025-01-24T09:10:42.51+00:00

Hi @Katherine Bayona

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to fix and troubleshoot 503 errors in Azure Cosmos DB with substatus code of 20003

1 answer

Your answer