Add Azure Cosmos DB CDC source to an eventstream

This article shows you how to add an Azure Cosmos DB (Azure Cosmos DB for NoSQL) Change Data Capture source to an eventstream.

The Azure Cosmos DB Change Data Capture (CDC) source connector for Microsoft Fabric event streams lets you capture a snapshot of the current data in an Azure Cosmos DB database. The connector then monitors and records any future row-level changes to this data. Once the changes are captured in the eventstream, you can process this CDC data in real-time and send it to different destinations within Fabric for further processing or analysis.

Note

This source is not supported in the following regions of your workspace capacity: West US3, Switzerland West.

Prerequisites

  • Access to a workspace in the Fabric capacity license mode (or) the Trial license mode with Contributor or higher permissions.
  • Access to an Azure Cosmos DB for NoSQL account and database.
  • Your Azure Cosmos DB for NoSQL database must be publicly accessible and not be behind a firewall or secured in a virtual network.
  • If you don't have an eventstream, create an eventstream.

Get connection details from the Azure portal

The labels for the items you need to collect from the Azure portal are shown in the following steps. You always need the endpoint URI, in a format like https://<account>.<api>.azure.com:<port>/, the Primary Key, and the Database name and item IDs you want to collect data for.

Note

Azure Cosmos DB for NoSQL CDC is using the Latest Version Mode of Azure Cosmos DB Change Feed. It captures the changes to records in the latest version. Note that Deletions are't captured with this mode.

  1. On the Azure portal page for your Azure Cosmos DB account, select Keys under Settings in the left navigation.

  2. On the Keys page, copy the URI and Primary key values to use for setting up the eventstream connection.

    A screenshot of the URI and Primary key on the Azure Cosmos DB Keys page in the Azure portal.

  3. On the Azure portal Overview page for your Azure Cosmos DB account, note the Database and item ID you want to collect data for.

    A screenshot of the Containers listing for an Azure Cosmos DB NoSQL API account.

Note

The maximum number of sources and destinations for one eventstream is 11.

Add Azure Cosmos DB (CDC) as a source

If you haven't added any source to your eventstream yet, select Use external source tile.

Screenshot that shows the selection of Use external source tile.

If you're adding the source to an already published eventstream, switch to Edit mode, select Add source on the ribbon, and then select External sources.

Screenshot that shows the selection of Add source to External sources menu.

On the Select a data source page, search for and select Connect on the Azure Event Hubs tile.

Screenshot that shows the selection of Azure Cosmos DB (CDC) as the source type in the Select a data source wizard.

Configure and connect to Azure Cosmos DB (CDC)

  1. On the Connect screen, under Connection, select New connection to create a cloud connection linking to your Azure Cosmos DB database.

    Screenshot that shows the Connect page with the New connection link selected.

  2. On the Connection settings screen, enter the following information:

    • Cosmos DB Endpoint: Enter the URI or Endpoint for your Cosmos DB account that you copied from the Azure portal.
    • Connection name: Automatically generated, or you can enter a new name for this connection.
    • Account key: Enter the Primary Key for your Azure Cosmos DB account that you copied from the Azure portal.

    A screenshot of the Connection settings for the Azure Cosmos DB CDC source.

  3. Select Connect.

  4. Provide the following information for your Azure Cosmos DB resources, and then select Next.

    • Container ID: Enter the name of the Azure Cosmos DB container or table you want to connect to.

    • Database: Enter the name of your Azure Cosmos DB database.

    • Offset policy: Select whether to start reading Earliest or Latest offsets if there's no commit.

      You can also change the Source name in the Stream details section on the right by selecting the Pencil button.

      A screenshot of the connection details for the Azure Cosmos DB CDC source.

  5. On the Review + connect page, review the summary, and then select Add.

    Screenshot that shows the Review + connect page for the Azure Cosmos DB CDC source.

View updated eventstream

  1. You see the Azure Cosmos DB (CDC) source added to your eventstream in Edit mode. Select Publish to publish the changes and begin streaming Azure Cosmos DB CDC data to the eventstream.

    A screenshot of the added Azure Cosmos DB CDC source in Edit mode with the Publish button highlighted.

  2. You see the eventstream in Live mode. Select Edit on the ribbon to get back to the Edit mode to update the eventstream.

    A screenshot of the published eventstream with Azure Cosmos DB source in Live View.

Other connectors: