How to change networking to connect IoTHub to CosmosDB in private subnet

Question

Hey community!

The question:

How do I connect my Azure IoTHub to my CosmosDB in private subnet? How I can change the CosmosDB networking/firewall settings to accept traffic coming from a (public) IoTHub service endpoint?

Brief description:

1. I have an IoTHub in which I host IoT devices for which the networking settings are set to allow connections from selected networks. For this I want a custom endpoint for routing to a CosmosDB container using a system-managed identity.
2. I also have an Azure CosmosDB account for which I want to limit the access so that the IoTHub can write messages to a container. The networking settings are set to selected to only allows access from a VNET.
   1. The IoTHub has the CosmosDB Data Contributor role assigned for this account.
3. Lastly, I have a VNET in which I have a private subnet, to which 2 Private Endpoints are connected (1 IoTHub, 1 CosmosDB). Additionally, Service Endpoints for CosmosDB, EventHub and ServiceBus are enabled.

When I try to connect the IoTHub custom endpoint routing I get the following error:

Cosmos DB authentication failed. Check whether primary Key or secondary Key is correct. ContainerName: [container_name], DatabaseName: [database-name], endpointName: [routing_endpoint_name], exceptionMessage: Response status code does not indicate success: Forbidden (403); Substatus: 0; ActivityId: [redacted]; Reason: (Request originated from VNET through service endpoint. This is blocked by your Cosmos DB account firewall settings. More info: https://aka.ms/cosmosdb-tsg-forbidden

Now I understand that the traffic is going through the service endpoint, but what I don't know is how to change the CosmosDB networking/firewall settings to accept messages from the IoTHub.

This connection works if the networking settings of the CosmosDB are set to public or selected networks with a whitelist for 0.0.0.0.

I know that some of the configurations mentioned are quite obvious, just wanted to paint a complete picture.

Hopefully you can help me out! Thanks!

Answer 1

Hello Quincy

Unless specific firewall rules have been set, adding a private endpoint makes your Azure Cosmos DB account accessible through private endpoints only. This means that the Azure Cosmos DB account could be reached from public traffic after it's created and before a private endpoint gets added.

https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-configure-private-endpoints?tabs=arm-bicep#blocking-public-network-access-during-account-creation

To fix 403 error You need to Configure the service endpoint for the Azure virtual network and subnet. You can do it in Cosmos DB settings' Firewall and virtual networks option.

1. Select Firewalls and virtual networks from the settings menu, and choose to allow access from Selected networks.
2. To grant access to an existing virtual network's subnet, under Virtual networks, select Add existing Azure virtual network.
3. Select the Subscription from which you want to add an Azure virtual network. Select the Azure Virtual networks and Subnets that you want to provide access to your Azure Cosmos DB account. Next, select Enable to enable selected networks with service endpoints for "Microsoft.AzureCosmosDB". When it's complete, select Add. Refer below image.
4. After the Azure Cosmos DB account is enabled for access from a virtual network, it will allow traffic from only this chosen subnet. The virtual network and subnet that you added should appear as shown in the following screenshot: https://stackoverflow.com/questions/71639169/request-originated-from-vnet-through-service-endpoint-this-is-blocked-by-your-c

The following situations and outcomes are possible when you use Private Link in combination with firewall rules:

• If you don't configure any firewall rules, then by default, all traffic can access an Azure Cosmos DB account.
• If you configure public traffic or a service endpoint and you create private endpoints, then different types of incoming traffic are authorized by the corresponding type of firewall rule. If a private endpoint is configured in a subnet where service endpoint is also configured:
  • traffic to the database account mapped by the private endpoint is routed via private endpoint,
  • traffic to other database accounts from the subnet is routed via service endpoint.
• If you don't configure any public traffic or service endpoint and you create private endpoints, then the Azure Cosmos DB account is accessible only through the private endpoints. If you don't configure public traffic or a service endpoint, after all approved private endpoints are rejected or deleted, the account is open to the entire network unless PublicNetworkAccess is set to Disabled.

https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-forbidden