Add a lakehouse destination to an eventstream

This article shows you how to add a lakehouse as a destination to an eventstream in Microsoft Fabric event streams. For optimized streaming performance and real-time querying, consider streaming data to an Eventhouse with add Eventhouse destination to an eventstream and then enable Eventhouse OneLake Availability.

Note

Enhanced capabilities are enabled by default when you create eventstreams now. If you have eventstreams that were created using standard capabilities, those eventstreams will continue to work. You can still edit and use them as usual. We recommend that you create a new eventstream to replace standard eventstreams so that you can take advantage of additional capabilities and benefits of enhanced eventstreams.

Important

There is schema enforcement for writing data into a lakehouse destination table. All new writes to the table must be compatible with the target table's schema at write time, ensuring data quality.

When output is written to a new delta table, the table schema is created based on the first record. All records of the output data are projected onto the schema of the existing table.

If the incoming data has columns that aren't in the existing table schema, the extra columns aren't included in the data written to the table. Likewise, if the incoming data is missing columns that are in the existing table schema, the missing columns write to the table with the values set to null.

Prerequisites

• Access to a workspace in the Fabric capacity license mode (or) the Trial license mode with Contributor or higher permissions.
• Access to the workspace where your lakehouse is located with Contributor or higher permissions.

Note

The maximum number of sources and destinations for one eventstream is 11.

Add a lakehouse as a destination

To add a lakehouse destination to a default or derived eventstream, follow these steps.

1. In Edit mode for your eventstream, select Add destination on the ribbon and select Lakehouse from the dropdown list.

   

2. Connect the lakehouse node to your stream node or operator.

3. On the Lakehouse configuration screen, complete the following information:

   1. Enter a Destination name.
   2. Select the Workspace that contains your lakehouse.
   3. Select an existing Lakehouse from the workspace you specified.
   4. Select an existing Delta table, or create a new one to receive data.
   5. Select the Input data format that is sent to your lakehouse. The supported data formats are JSON, Avro, and CSV (with header).

   

4. Select Advanced.

5. Two ingestion modes are available for a lakehouse destination. Based on your scenario, configure these modes to optimize how Fabric event streams writes to the lakehouse.

   • Minimum rows is the minimum number of rows that the lakehouse ingests in a single file. The minimum is 1 row, and the maximum is 2 million rows per file. The smaller the minimum number of rows, the more files the lakehouse creates during ingestion.

   • Maximum duration is the maximum duration that the lakehouse takes to ingest a single file. The minimum is 1 minute and maximum is 2 hours. The longer the duration, the more rows are ingested in a file.

   

6. Select Save.

7. To implement the newly added lakehouse destination, select Publish.

   

Once you complete these steps, the lakehouse destination is available for visualization in Live view. In the Details pane, you can select the Optimize table in notebook shortcut to launch an Apache Spark job within a Notebook, which consolidates the small streaming files within the target lakehouse table.

Related content

To learn how to add other destinations to an eventstream, see the following articles:

• Route events to destinations
• Custom app
• Derived stream
• Eventhouse
• Fabric Activator
• Create an eventstream

Prerequisites

Before you start, you must complete the following prerequisites:

• Access to a workspace in the Fabric capacity license mode (or) the Trial license mode with Contributor or higher permissions.
• Get access to a workspace with Contributor or above permissions where your lakehouse is located.

Note

The maximum number of sources and destinations for one eventstream is 11.

Add a lakehouse as a destination

If you have a lakehouse created in your workspace, follow these steps to add the lakehouse to your eventstream as a destination:

1. Select New destination on the ribbon or "+" in the main editor canvas and then select Lakehouse. The Lakehouse destination configuration screen appears.

2. Enter a name for the eventstream destination and complete the information about your lakehouse.

   

   1. Lakehouse: Select an existing lakehouse from the workspace you specified.

   2. Delta table: Select an existing delta table or create a new one to receive data.

      Note

      When writing data into the lakehouse table, there is Schema enforcement. This means all new writes to a table must be compatible with the target table's schema at write time, ensuring data quality.

      All records of the output data are projected onto the schema of the existing table. When writing the output to a new delta table, the table schema is created based on the first record. If the incoming data has an additional column compared to the existing table schema, it writes to the table without including the extra column. Conversely, if the incoming data is missing a column compared to the existing table schema, it writes to the table with the column's value set to null.

   3. Input data format: Select the format for the data (input data) that is sent to your lakehouse.

      Note

      The supported input event data formats are JSON, Avro, and CSV (with header).

   4. Event processing: You can use the event processing editor to specify how the data should be processed before sending it to your lakehouse. Select Open event processor to open the event processing editor. To learn more about real-time processing using the event processor, see Process event data with event processor editor. When you're done with the editor, select Done to return to the Lakehouse destination configuration screen.

      

3. Two ingestion modes are available for a lakehouse destination. Select one of these modes to optimize how the Fabric event streams feature writes to the lakehouse based on your scenario.

   1. Rows per file – The minimum number of rows that Lakehouse ingests in a single file. The smaller the minimum number of rows, the more files Lakehouse creates during ingestion. Minimum is 1 row. Maximum is 2M rows per file.

   2. Duration – The maximum duration that Lakehouse would take to ingest a single file. The longer the duration, more rows are ingested in a file. Minimum is 1 minute and maximum is 2 hours.

      

4. Select Add to add the lakehouse destination.

5. Table optimization shortcut available inside the lakehouse destination. This solution facilitates you by launching a Spark job within a Notebook, which consolidates these small streaming files within the target Lakehouse table.

   

6. A lakehouse destination appears on the canvas, with a spinning status indicator. The system takes a few minutes to change the status to Active.

   

Manage a destination

Edit/remove: You can edit or remove an eventstream destination either through the navigation pane or canvas.

When you select Edit, the edit pane opens in the right side of the main editor. You can modify the configuration as you wish, including the event transformation logic through the event processor editor.

Related content

To learn how to add other destinations to an eventstream, see the following articles:

• Route events to destinations
• Custom app
• KQL Database
• Fabric Activator
• Create an eventstream