Edit

Share via

Add Amazon MSK Kafka source to an eventstream

  • Article

This article shows you how to add an Amazon MSK Kafka source to an eventstream. 

Amazon MSK Kafka is a fully managed Kafka service that simplifies the setup, scaling, and management. By integrating Amazon MSK Kafka as a source within your eventstream, you can seamlessly bring the real-time events from your MSK Kafka and process it before routing them to multiple destinations within Fabric. 

Note

This source is not supported in the following regions of your workspace capacity: West US3, Switzerland West.

Prerequisites

  • Access to a workspace in the Fabric capacity license mode (or) the Trial license mode with Contributor or higher permissions.
  • An Amazon MSK Kafka cluster in active status.
  • Your Amazon MSK Kafka cluster must be publicly accessible and not be behind a firewall or secured in a virtual network. 

Note

The maximum number of sources and destinations for one eventstream is 11.

Add Amazon MSK Kafka as a source 

  1. In Fabric Real-Time Intelligence, select Eventstream to create a new eventstream.
  2. On the next screen, select Add external source.

Configure and connect to Amazon MSK Kafka

  1. On the Data sources page, select Amazon MSK Kafka.

    Screenshot that shows the selection of Amazon MSK Kafka as the source type in the Get events wizard.

  2. On the Connect page, select New connection.

    Screenshot that shows the selection of the New connection link on the Connect page of the Get events wizard.

  3. In the Connection settings section, for Bootstrap Server, enter the public endpoint of your Kafka cluster.

    Screenshot that shows the selection of the Bootstrap server field on the Connect page of the Get events wizard.

    To get the public endpoint:

    Screenshot that shows the public endpoint of Amazon MSK cluster.

  4. In the Connection credentials section, If you have an existing connection to the Amazon MSK Kafka cluster, select it from the drop-down list for Connection. Otherwise, follow these steps:

    1. For Connection name, enter a name for the connection.
    2. For Authentication kind, confirm that API Key is selected.
    3. For Key and Secret, enter API key and key Secret for Amazon MSK Kafka cluster.

  5. Select Connect. 

  6. Now, on the Connect page, follow these steps.

    1. For Topic, enter the Kafka topic.

    2. For Consumer group, enter the consumer group of your Kafka cluster. This field provides you with a dedicated consumer group for getting events.

    3. Select Reset auto offset to specify where to start reading offsets if there's no commit.

    4. For Security protocol, the default value is SASL_SSL. The default SASL mechanism is SCRAM-SHA-512 and cannot be changed.

      Screenshot that shows the first page of the Amazon MSK Kafka connection settings.

  7. Select Next. On the Review + connect screen, review the summary, and then select Connect.

View updated eventstream

You can see the Amazon MSK Kafka source added to your eventstream in Edit mode. 

Screenshot that shows Amazon MSK Kafka source in Edit view.

After you complete these steps, the Amazon MSK Kafka source is available for visualization in Live view.

Screenshot that shows Amazon MSK Kafka source in Live view.

Related content

Other connectors: