CosmosDB cannot read out documents with URL encoded IDs

Question

CosmosDB cannot read out documents with URL encoded IDs

Mikkel 10

We had a Cosmos DB account and in one of the containers some of the documents have URL encoded IDs, e.g.: rfq_123_%28test%29 meaning "rfq_123_(test)". I'm no fan of url encoded IDs, but I'm just supporting the project and it's too late to change.
We could read this document via Data Explorer in the Azure Portal by clicking on the document.

The account was destroyed and restored (point in time) to a temporary account as part of our disaster recovery plan. The restored account also works as expected.
We then provisioned a new Cosmos DB account via Terraform and used a az cli based copy script to copy data from the temporary account to the new account.

After the data was copied, however we could no longer read out documents with url encoded IDs, so when clicking on said document in the new account, we got the following error:

Failed to read item rfq_123_%28test%29: The input authorization token can't serve the request. The wrong key is being used or the expected payload is not built as per the protocol. For more info: https://aka.ms/cosmosdb-tsg-unauthorized. Server used the following payload to sign: 'get docs dbs/master/colls/test/docs/rfq_123_(test) thu, 25 jul 2024 11:45:44 gmt

We can still query the document and get the document content:

SELECT * FROM c WHERE c.id = 'rfq_123_%28test%29'

I have diff'ed all settings and configurations between the restored (working) account and the new account with copied data. They are identical apart from the restored account having some irrelevant restore timestamp related settings.

Now to the real joker: After about 22-24 hours, the new account suddenly started to work again. Documents with url encoded IDs could be read from the Data Explorer in the Portal.
I felt a bit uncomfortable about that, so I tried manually creating a new Cosmos DB account, added a container with the same settings as my real account and manually inserted a document with a url encoded ID. Lo and behold: The document is inserted, but cannot be read via the Portal or the SDK.

This definitely seems like a critical Cosmos DB bug and any support would be appreciated.

To reproduce:

Create a new Cosmos DB account.
Add a database and container.
Insert dummy document with a url encoded ID, e.g. rfq_123_%28test%29
Try to read the document from the Data Explorer.
This will result in an error message in the portal.

Oury Ba-MSFT 20,446 Reputation points Microsoft Employee

2024-07-25T22:31:40.84+00:00

@Mikkel Thank you for reaching out.

Thanks a lot for pointing this out. It’s a known issue.

As a workaround, customers can correct the IDs with issues programmatically using the SDKs.

Customers should ideally not create docs with special characters in the id, if they want to be able to view them from the portal.

Regards,

Oury
Mikkel 10 Reputation points

2024-07-29T07:31:26.27+00:00

@Oury Ba-MSFT Thank you for your comment.

But why does it then work for some of our Cosmos DB accounts and databases, while not for others? They have identical settings and data.

I went through the Cosmos DB documentation. URL encoding of document IDs are not mentioned and does not seem to be on the list of restricted special characters. The documentation states and I quote: "The following characters are restricted and cannot be used in the Id property: '/', '\', '?', '#'".

Can your Cosmos DB team please look more into this? We have a system in production with document IDs containing url encoded characters, but none of the restricted special characters, and I will have to be able to document why we potentially have to undertake a rather large migration task, if what you claim about url encoded document IDs is correct.

Thank you in advance.

Best
Mikkel
Oury Ba-MSFT 20,446 Reputation points Microsoft Employee

2024-10-18T16:08:28.7+00:00

@Mikkel Sorry for the delay in response to your follow up question.

Some special characters are not supported in older Gateway versions (Routing Gateway) while they would work in newer Gateway versions (Compute Gateway) - if that is causing the difference in behavior (we would be able to check from the account names) it should be possible to switch the accounts to the new version of the Gateway. For that You might need to create a support case so we can do it from the back end.

Regards,

Oury
Mo 0 Reputation points

2025-03-11T12:30:59.2266667+00:00

This bug really needs fixing. We're specifically URL encoding our IDs because certain characters aren't allowed in cosmos keys, and that's the safest way to deal with that, but then it turns out the client library doesn't know what to do with those and decodes them into url fragments

Your answer

Oury Ba-MSFT 20,446 Reputation points Microsoft Employee

2024-07-25T22:31:40.84+00:00

@Mikkel Thank you for reaching out.

Thanks a lot for pointing this out. It’s a known issue.

As a workaround, customers can correct the IDs with issues programmatically using the SDKs.

Customers should ideally not create docs with special characters in the id, if they want to be able to view them from the portal.

Regards,

Oury
Mikkel 10 Reputation points

2024-07-29T07:31:26.27+00:00

@Oury Ba-MSFT Thank you for your comment.

But why does it then work for some of our Cosmos DB accounts and databases, while not for others? They have identical settings and data.

I went through the Cosmos DB documentation. URL encoding of document IDs are not mentioned and does not seem to be on the list of restricted special characters. The documentation states and I quote: "The following characters are restricted and cannot be used in the Id property: '/', '\', '?', '#'".

Can your Cosmos DB team please look more into this? We have a system in production with document IDs containing url encoded characters, but none of the restricted special characters, and I will have to be able to document why we potentially have to undertake a rather large migration task, if what you claim about url encoded document IDs is correct.

Thank you in advance.

Best
Mikkel
Oury Ba-MSFT 20,446 Reputation points Microsoft Employee

2024-10-18T16:08:28.7+00:00

@Mikkel Sorry for the delay in response to your follow up question.

Some special characters are not supported in older Gateway versions (Routing Gateway) while they would work in newer Gateway versions (Compute Gateway) - if that is causing the difference in behavior (we would be able to check from the account names) it should be possible to switch the accounts to the new version of the Gateway. For that You might need to create a support case so we can do it from the back end.

Regards,

Oury
Mo 0 Reputation points

2025-03-11T12:30:59.2266667+00:00

This bug really needs fixing. We're specifically URL encoding our IDs because certain characters aren't allowed in cosmos keys, and that's the safest way to deal with that, but then it turns out the client library doesn't know what to do with those and decodes them into url fragments

Share via

CosmosDB cannot read out documents with URL encoded IDs

Your answer