Edit

Share via

Add Apache Kafka as source in Real-Time hub (preview)

  • Article

This article describes how to add Apache Kafka as an event source in Fabric Real-Time hub.

Prerequisites

  • Access to a workspace in the Fabric capacity license mode (or) the Trial license mode with Contributor or higher permissions.
  • An Apache Kafka cluster running.
  • Your Apache Kafka must be publicly accessible and not be behind a firewall or secured in a virtual network. 

Data sources page

  1. Sign in to Microsoft Fabric.

  2. Select Real-Time on the left navigation bar.

  3. On the Real-Time hub page, select + Data sources under Connect to on the left navigation menu. You can also get to the Data sources page from All data streams or My data streams pages by selecting the + Connect data source button in the top-right corner.

    Screenshot that shows how to launch Connect to data source experience.

Add Apache Kafka as a source

  1. On the Select a data source page, select Apache Kafka.

    Screenshot that shows the selection of Apache Kafka as the source type in the Get events wizard.

  2. On the Connect page, select New connection.

    Screenshot that shows the selection of the New connection link on the Connect page of the Get events wizard.

  3. In the Connection settings section, for Bootstrap Server, enter your Apache Kafka server address.

    Screenshot that shows the selection of the Bootstrap server field on the Connect page of the Get events wizard.

  4. In the Connection credentials section, If you have an existing connection to the Apache Kafka cluster, select it from the drop-down list for Connection. Otherwise, follow these steps:

    1. For Connection name, enter a name for the connection.
    2. For Authentication kind, confirm that API Key is selected.
    3. For Key and Secret, enter API key and key Secret.

  5. Select Connect. 

  6. Now, on the Connect page, follow these steps.

    1. For Topic, enter the Kafka topic.

    2. For Consumer group, enter the consumer group of your Apache Kafka cluster. This field provides you with a dedicated consumer group for getting events.

    3. Select Reset auto offset to specify where to start reading offsets if there's no commit.

    4. For Security protocol, the default value is SASL_PLAINTEXT.

      Note

      The Apache Kafka source currently supports only unencrypted data transmission (SASL_PLAINTEXT and PLAINTEXT) between your Apache Kafka cluster and Eventstream. Support for encrypted data transmission via SSL will be available soon.

    5. The default SASL mechanism is typically PLAIN, unless configured otherwise. You can select the SCRAM-SHA-256 or SCRAM-SHA-512 mechanism that suits your security requirements.

      Screenshot that shows the first page of the Apache Kafka connection settings.

  7. Select Next. On the Review and create screen, review the summary, and then select Add.

View data stream details

  1. On the Review + connect page, if you select Open eventstream, the wizard opens the eventstream that it created for you with the selected Apache Kafka source. To close the wizard, select Close at the bottom of the page.

  2. In Real-Time hub, switch to the Data streams tab of Real-Time hub. Refresh the page. You should see the data stream created for you.

    For detailed steps, see View details of data streams in Fabric Real-Time hub.

Related content

To learn about consuming data streams, see the following articles: