What's new and planned for Real-Time Intelligence in Microsoft Fabric
Important
The release plans describe functionality that may or may not have been released yet. The delivery timelines and projected functionality may change or may not ship. Refer to Microsoft policy for more information.
Real-Time Intelligence enables data professionals, analysts, and business users to ingest, process, analyze, transform, visualize and act on large, time-sensitive and highly granular data to help organizations make faster and more informed business decisions. It is an end-to-end experience that enables seamless handling of real-time data without the need to land it first. Real-Time Intelligence builds on the existing Real-Time Analytics and Data Activator capabilities and offers new features making it even easier for users of all skill levels to get the most from their real-time data. This includes ingesting streaming data with high granularity, dynamically transform streaming data, query data in real-time for instant insights, and trigger actions like alerting a production manager when equipment is overheating or rerunning jobs when data pipelines fail.
Even though it’s called “real-time”, your data doesn’t have to be flowing at high rates and volumes to get true business value. Real-Time Intelligence gives you event-driven, rather than schedule-driven solutions, ensuring you can make the best decisions on time.
Behind this powerful workload is the Real-Time hub, a single place to discover, manage and use event steaming data from Fabric, other Microsoft data sources, and external data sources. Just like the OneLake data hub makes it easy to discover, manage, and use the data at rest, the Real-Time hub can help you do the same for data in motion. All events that flow through the Real-Time hub can be easily transformed and routed to any Fabric data stores and can create new streams that can be discovered and consumed.
For more information, see the Real-Time Intelligence documentation and visit the blog.
The items available in Real-Time Intelligence are:
Real-Time hub, a single place to discover, manage and use event steaming data from Fabric, other Microsoft data sources, and external data sources
Eventstreams for capturing, transforming, and routing real-time events to various destinations with a no-code experience.
Eventhouse and KQL database are the ideal analytics engine to process data in motion. They're tailored to time-based, streaming events with structured, semi structured, and unstructured data. This data is automatically indexed and partitioned based on ingestion time, giving you incredibly fast and complex analytic querying capabilities on high-granularity data. Data stored in event houses can be made available in OneLake for consumption by other Fabric experiences.
KQL queryset to run queries, view, and customize query results on data. The KQL queryset allows you to save queries for future use, export, and share queries with others. It includes the option to generate a Power BI report.
Real-time dashboards, which contains a collection of tiles. Each tile has an underlying query and a visual representation perfect for data exploration, monitoring, and forensics.
Activator that integrates with Real-Time hub, eventstreams, real-time dashboards and KQL querysets, making it seamless to take action on any patterns or changes in real-time data.
Investment areas
CI/CD support for Eventstream items
Estimated release timeline: Q4 2024
Release Type: Public preview
Customers can easily version and deploy their eventstreams across development, testing, and production workspaces using the integrated Git integration and deployment experience in the Fabric platform.
Real-Time Dashboards General Availability
Estimated release timeline: Q4 2024
Release Type: General availability
General availability of Fabric Real-Time Dashboards
Data Exploration General Availability
Estimated release timeline: Q4 2024
Release Type: General availability
General availability of no-code data exploration experiences for Real-Time Intelligence, including the ability to explore real-time dashboards tile data and query results.
Copilot for Real-Time Intelligence
Estimated release timeline: Q4 2024
Release Type: General availability
Copilot for Real-Time Intelligence is an advanced AI tool designed to help you explore your data and extract valuable insights. You can input questions about your data, which are then automatically translated into Kusto Query Language (KQL) queries. Copilot streamlines the process of analyzing data for both experienced KQL users and citizen data scientists.
New streaming connector for Azure Service Bus
Estimated release timeline: Q4 2024
Release Type: Public preview
Customers can leverage new connector to bring the streaming events from Azure Service Bus to Eventstream.
Activator performance & scale improvements
Estimated release timeline: Q4 2024
Release Type: General availability
Activator items will support up to 1000 events per second per stream, and activations typically happen within 30 seconds of the event being received.
Activator alerts on Power BI, KQL Queryset and Real-Time Dashboard
Estimated release timeline: Q4 2024
Release Type: General availability
Activator will support monitoring data from Power BI reports, Querysets, and Real-Time Dashboards. You set up an alert and configure it in a side pane.
"Get Events" from Activator
Estimated release timeline: Q4 2024
Release Type: General availability
From within an activator item, you choose ‘Get data’ to pull events from Real-Time Hub. You get search and catalog capabilities over all streams in your organization, and can connect to external sources with connectors to streams, change data capture feeds and more.
Fabric item job actions
Estimated release timeline: Q4 2024
Release Type: General availability
Activator lets you start Fabric items as an action. For example, you can run a Fabric pipeline when files are created in Azure data lake storage. Fabric pipelines and Spark notebooks are supported.
Maximum number of rule actions
Estimated release timeline: Q4 2024
Release Type: General availability
To avoid overloading messaging systems and generating too much noise, Activator has limits on how many times a rule can send an email/Teams message or trigger a Power Automate flow. This stops users being flooded with messages.
Fabric Monitoring integration
Estimated release timeline: Q4 2024
Release Type: Public preview
Fabric Workspace Monitoring is designed to provide a seamless and consistent monitoring experience with end-to-end visibility across all Fabric items. Workspace Monitoring is based on the Real-time Intelligence Eventhouse KQL database. Once Fabric Monitoring is enabled, a KQL database is created to store all the workspace items event logs. KQL databases are ideal for time series logs and metrics monitoring solutions.
Power BI alert management
Estimated release timeline: Q4 2024
Release Type: General availability
Users will be able to see and manage all alerts configured with Activator alerts on their current Power BI report. Choosing ‘Set alert’ for a report will present a list of all the alerts that you set up on that report, giving you the option to add new ones, start or stop rules, or delete them if they’re no longer necessary.
Migration from Synapse Data Explorer pools to Fabric Eventhouse
Estimated release timeline: Q4 2024
Release Type: Public preview
Seamless migration from Synapse Data Explorer pools to Fabric Eventhouse.
Query Acceleration over OneLake Delta Tables
Estimated release timeline: Q4 2024
Release Type: Public preview
Running queries on shortcuts is typically slow due to various factors such as network calls to fetch data from storage, the absence of indexes, and more.
Customers who are used to running performant queries on top of ingested data in Eventhouse, will be able to gain similar performance by caching Delta tables with query acceleration .
Additional Fabric system events
Estimated release timeline: Q4 2024
Release Type: Public preview
More Fabric Event categories will be available for use in Real-Time hub, namely Fabric Job events, OneLake events
Create Real time dashboards with Copilot
Estimated release timeline: Q4 2024
Release Type: Public preview
Auto create Real-Time Dashboard for KQL tables, including metadata tiles and insights tiles generated by Copilot
Eventstream connector sources GA
Estimated release timeline: Q4 2024
Release Type: General availability
Connector sources to be GA. Connector source list:
- Azure SQL Database Change Data Capture (CDC)
- SQL Server on VM DB (CDC)
- Azure SQL Managed Instance CDC
- PostgreSQL Database CDC
- MySQL Database CDC
- Azure Cosmos DB CDC
- Google Cloud Pub/Sub
- Amazon Kinesis Data Streams
- Confluent Cloud Kafka
- Amazon MSK Kafka
Data insights (metrics) on connector sources in Eventstream
Estimated release timeline: Q4 2024
Release Type: General availability
Support the data insights (metrics) on connector sources
Runtime logs on connector sources in Eventstream
Estimated release timeline: Q4 2024
Release Type: General availability
Support the runtime logs on connector sources
Eventstream connector source pausing and resuming
Estimated release timeline: Q4 2024
Release Type: General availability
Support to pause and resume the connector sources.
Processing and routing events to Data Activator destination
Estimated release timeline: Q4 2024
Release Type: Public preview
Support to process the events before routing the events to data activator destination.
Entra ID authentication support for custom endpoint in Eventstream
Estimated release timeline: Q4 2024
Release Type: General availability
Entra ID authentication support in custom endpoint
New Fabric events sources: job events and onelake item events
Estimated release timeline: Q4 2024
Release Type: Public preview
New Fabric events sources: job events and onelake item events
Data preview on new sources: CDC source in Eventstream
Estimated release timeline: Q4 2024
Release Type: General availability
Support to preview the data on the database CDC connector sources inside Eventstream.
Public (REST) APIs support for Eventstream items
Estimated release timeline: Q4 2024
Release Type: Public preview
Eventstream Public APIs enable the e2e automated interaction with Eventstream item, and the integration of Eventstream item with external applications.
Eventhouse Platform Monitoring
Estimated release timeline: Q4 2024
Release Type: Public preview
Access Eventhouse logs and metrics to fine tune performance, troubleshoot issues and identify anomalies. The Eventhouse will report on Query logs, Ingestion metrics, and Ingestion failure events. The Eventhouse event and metric logs will be stored in a read only KQL database. Users will be able to analyze the events using KQL Querysets and Real time Dashboards.
Copilot assisted Real-Time dashboard creation for KQL table in Real-Time Hub
Estimated release timeline: Q4 2024
Release Type: Public preview
Customers can select a KQL table and generate a real-time dashboard with copilot.
Rich sample experience in Real-Time hub landing page
Estimated release timeline: Q4 2024
Release Type: General availability
When users first time come to Real-TIme hub, provide rich samples for them to try out.
Simplified Eventhub connection to Eventstream
Estimated release timeline: Q4 2024
Release Type: General availability
We will autopopulate Eventhub information so that customers can connect to Eventhubs within several clicks.
KQL Database Entities Diagram
Estimated release timeline: Q4 2024
Release Type: Public preview
a visual representation of the KQL DB entities and their relations (such as tables, functions, materialized views, and more).
Evenhouse table no code creation and editing experience
Estimated release timeline: Q4 2024
Release Type: Public preview
Wizards to create/edit entities like functions, update policies, and materialized views.
Enhanced Eventstream GA: Content-based routing, derived streams and Edit/Live modes
Estimated release timeline: Q4 2024
Release Type: General availability
Enhanced Eventstream GA with content-based routing, derived streams and Edit/Live modes
Explore data for KQL table in Real-Time hub
Estimated release timeline: Q4 2024
Release Type: General availability
Allow customers to use 'explore data' capability for KQL tables in Real-Time hub.
Eventstream to Activator events
Estimated release timeline: Q1 2025
Release Type: General availability
Activator can monitor and act on events ingested to Fabric through an Eventstream. This will reach General Availability in early 2025.
Support for parameters in triggering Fabric actions
Estimated release timeline: Q1 2025
Release Type: Public preview
Rules that take action by starting Fabric jobs (notebooks or pipelines) will be able to pass the value of properties in the activator model as parameters to the job.
Fabric job action parameters
Estimated release timeline: Q1 2025
Release Type: General availability
When starting a Fabric job as an Activator action, users can pass values as parameters to the job. For example, the values from an event could be passed to the notebook or pipeline’s parameters for further use or transformation.
Activator delay tolerance controls
Estimated release timeline: Q1 2025
Release Type: General availability
Streams of events often arrive out-of-order. When comparing the change between a new event and the previous one, customers ask for control over how long to wait before ignoring late arriving events. This affects the overall latency of the rule, but allows users the manage the tradeoff between speed and accuracy.
Event catalog integration
Estimated release timeline: Q1 2025
Release Type: Public preview
Real-Time hub expands capabilities to support managing event types, providing a catalog for schema management and discovery.
New Fabric events sources: capacity utilization events
Estimated release timeline: Q1 2025
Release Type: Public preview
New Fabric events sources: capacity utilization events
SQL native support in KQL querysets
Estimated release timeline: Q2 2025
This feature enables customers to use a native SQL editor to run SQL over KQL databases in a queryset, alongside using KQL. With this capability, customers are able to use the SQL editor’s native capabilities, such as syntax highlighting, suggestions, and more.
Shipped feature(s)
Real-Time Hub for all data in motion GA
Shipped (Q4 2024)
Real-Time hub is the single estate for all data-in-motion across your entire organization.
Get events experience in Real-Time Hub (change to explain value or remove for 18/24)
Shipped (Q4 2024)
Release Type: General availability
Whether data is coming from new or existing sources, streams, or available events, the Get Events experience allows users to connect to a wide range of sources directly from Real-Time hub, Eventstreams, Eventhouse and Data Activator.
Eventhouse ALM Support
Shipped (Q3 2024)
Eventhouse ALM Support to allow for automatic deployments via Git Integration, Deployment Pipelines, and API with Definition.
Eventstream integration with Managed Private Endpoint
Shipped (Q3 2024)
Release Type: Public preview
This integration with Managed Private Endpoints enables secure and private access to data sources behind a firewall or blocked from public internet access, allowing data to be streamed to Fabric Eventstream over outbound connections.
Eventhouse destination support Eventstreams
Shipped (Q3 2024)
Release Type: Public preview
Eventstreams will add support for Eventhouse as a destination.
New streaming connectors for Kafka public clusters and more instances of SQL Server DB CDC
Shipped (Q3 2024)
Release Type: Public preview
Additional connectors will be available through the Get Events experience, including Public Kafka clusters and SQL Server DB CDC.
Set Alerts on KQL Querysets
Shipped (Q3 2024)
Release Type: Public preview
Data Activator will be able to monitor the results of queries managed in a KQL Queryset, running the query on a regular schedule and acting on the results.
More trigger options
Shipped (Q3 2024)
Release Type: Public preview
To address further requests for monitoring different business conditions, Data Activator will support triggers that check date/time values, basic math functions etc.
Low latency triggers
Shipped (Q3 2024)
We hear in customer feedback that triggers need to react faster when the specified condition is met. Data Activator will support lower-latency triggers in scenarios with simple conditions and lower volumes of data.This feature will be Private Preview in this time frame.
Azure Synapse Link from Cosmos DB to Fabric Eventhouse
Shipped (Q3 2024)
Users can stream all changes from Cosmos DB to a Fabric Eventhouse KQL database, enabling large scale, powerful analytics via Eventstream Cosmos DB CDC.
- /fabric/real-time-intelligence/event-streams/add-source-azure-cosmos-db-change-data-capture
- /fabric/real-time-intelligence/get-data-eventstream
Multivariate Anomaly Detection
Shipped (Q3 2024)
The Azure Anomaly Detector multi variate technology will be natively available in Fabric. Train the detector using spark and notebooks or using Eventhouses, and use it to detect anomalies in real time as new data arrives.
Real time dashboard in Fabric OrgApps
Shipped (Q3 2024)
Org apps support for Real time dashboards
Real-Time Dashboard
Shipped (Q2 2024)
Release Type: Public preview
A dashboard is a collection of tiles, optionally organized in pages, where each tile has an underlying query and a visual representation. Natively export Kusto Query Language (KQL) queries to a dashboard as visuals and later modify their underlying queries and visual formatting as needed. In addition to ease of data exploration, this fully integrated dashboard experience provides improved query and visualization performance with light, flexible modeling on high granularity low latency data.
Content-based event routing to destinations in eventstreams
Shipped (Q2 2024)
Release Type: Public preview
Bringing the no-code event processor to the eventstream main canvas allows customers to route event data to their destinations based on event content, using event processing logic defined with the event processor.
Create actions and alerts with Data Activator
Shipped (Q2 2024)
This feature provides a low-code/no-code experience to drive actions and alerts from your KQL database data presented in a Real-Time Dashboard. Data Activator gives you a single place to define actionable patterns in your data. These patterns can range from simple thresholds (such as a value being exceeded) to more complex patterns over time (such a value trending down). When Data Activator detects an actionable pattern, it triggers an action. That action can be an email or a Teams alert to the relevant person in your organization. It can also trigger an automatic process, via a Power Automate flow or an action in one of your organization’s line-of-business apps.
Pause, stop, or restart eventstream flows
Shipped (Q2 2024)
This feature offers customers the capability to pause, stop, or restart real-time event flows, so they can iterate on transformation or event flow logic during the development phase.
Co-pilot for Real-Time Intelligence
Shipped (Q2 2024)
Release Type: Public preview
Co-pilot allows you to write queries in natural language and have them translated into Kusto Query Language (KQL). You can use Co-pilot to ask your how-to queries, explore your data in a KQL database, and create Kusto entities such as tables, functions, and materialized views.
Eventhouse
Shipped (Q2 2024)
Release Type: General availability
Eventhouse, a cutting-edge database workspace meticulously crafted to manage and store event-based data, is now officially available for general use. With Eventhouse, users can perform high-performance analysis of big data and real-time data querying, processing billions of events within seconds. The platform allows users to organize data into compartments (databases) within one logical item, facilitating efficient data management. Additionally, Eventhouse enables the sharing of compute and cache resources across databases, maximizing resource utilization.
Eventhouse OneLake Availability
Shipped (Q2 2024)
Release Type: General availability
Enabling data availability of Eventhouse in OneLake means that customers can enjoy the best of both worlds: they can query the data with high performance and low latency in their Eventhouse and query the same data in Delta Lake format via any other Fabric engines such as Power BI Direct Lake mode, Warehouse, Lakehouse, Notebooks, and more.
Create a database shortcut to another KQL Database
Shipped (Q2 2024)
Release Type: Public preview
A database shortcut in Eventhouse is an embedded reference to a source database. The database shortcut is attached in read-only mode, making it possible to view and run queries on the data that was ingested into the source KQL Database without ingesting it. This helps with data sharing scenarios where you can share data in-place either within teams, or even with external customers.
Support for AI Anomaly Detector
Shipped (Q2 2024)
Release Type: Public preview
AI Anomaly Detector algorithms are supported in Microsoft Fabric, allowing for real time scoring by KQL with inline Python in Real-Time Intelligence.
Eventhouse tenant level private endpoint support
Shipped (Q2 2024)
Release Type: Public preview
Customers can increase their network security by limiting access to Eventhouse at a tenant-level, from one or more virtual networks (VNets) via private links. This will prevent unauthorized access from public networks and only permit data plane operations from specific VNets.
New experience for data exploration from Real-Time Dashboards
Shipped (Q2 2024)
Release Type: Public preview
Directly from a real-time dashboard, users can refine their exploration using a user-friendly, form-like interface. This intuitive and dynamic experience is tailored for insights explorers craving insights based on real-time data. Add filters, create aggregations, and switch visualization types without writing queries to easily uncover insights.
Use Real-Time hub to Get Data in KQL Database in Eventhouse
Shipped (Q2 2024)
Release Type: Public preview
With Real-Time hub embedded in KQL Database experience, each user in the tenant can view and add streams which they have access and directly ingest it to a KQL Database table in Eventhouse.
Get data from Real-Time Hub within Data Activator
Shipped (Q2 2024)
Release Type: Public preview
Bring new events into reflex items by using the embedded Real-Time Hub browser.
Create triggers from Real-Time Hub
Shipped (Q2 2024)
Release Type: Public preview
Across the Real-Time Hub you'll see options to create new triggers and alerts on both business and system events.
Taking action through Fabric Items
Shipped (Q2 2024)
Release Type: Public preview
Reflex items will support starting Fabric item jobs as an action. You'll be able to kick off data pipelines and notebooks.
Set Alert on Real-Time Dashboards
Shipped (Q2 2024)
Release Type: Public preview
In addition to monitoring Power BI and Eventstream data, Data Activator will support periodic querying against KQL queries saved in Real-Time Dashboards. These queries can be used to return exceptions, or track results over time to detect data changes or threshold conditions.
New streaming connectors including Confluent Kafka, AWS Kinesis, PostgresSQL CDC and more
Shipped (Q2 2024)
Release Type: Public preview
Beyond Microsoft sources, users can now stream data from other platforms like Google Cloud, Amazon Kinesis, Database change data capture streams, etc. using our new messaging connectors. Our messaging connectors are powered by Kafka connect with Camel Kafka connectors that offer a pluggable, declarative data integration framework, and flexibility for diverse source connections on popular data platforms, as well as the Debezium that could connect databases for fetching the Change Data Capture (CDC) streams.
New sources of discrete events from Fabric system events and Azure storage events
Shipped (Q2 2024)
Release Type: Public preview
Discrete events (a.k.a. notification events) are significant (important) facts that have happened, and that trigger state changes or actions in the downstream consumer applications (subscribers). Examples include events like OrderPlaced in order system, blob storage changes in Azure sources, or item changes in Fabric system.
Source from Real-Time Hub in Enhanced Eventstream
Shipped (Q2 2024)
Release Type: Public preview
The Eventstream homepage makes it even easier to bring in data. By clicking on the “Add external source”, you will find these sources in the Get events wizard that helps you to set up the source in a few steps. After you add the source to your eventstream, you can publish it to stream the data into your eventstream. Using Eventstream with discrete sources to turn events into streams for more analysis. You can send the streams to different Fabric data destinations, like Lakehouse and KQL Database.
Eventstream Edit and Live modes
Shipped (Q2 2024)
Release Type: Public preview
Eventstream offers two distinct modes, Edit and Live, to provide flexibility and control over the development process of your eventstream. If you create a new Eventstream with Enhanced Capabilities enabled, you can modify it in an Edit mode. Here, you can design stream processing operations for your data streams using a no-code editor. Once you complete the editing, you can publish your Eventstream and visualize how it starts streaming and processing data in Live mode.
Default and Derived Streams
Shipped (Q2 2024)
Release Type: Public preview
Multiple streams are listed within the Real-Time Hub including Default and Derived streams. Default stream are automatically generated when a streaming source is added to Eventstream. Default stream captures raw event data directly from the source, ready for transformation or analysis. erived stream: A specialized stream that users can create as a destination within Eventstream. Derived stream can be created after a series of operations such as filtering and aggregating, and then it’s ready for further consumption or analysis by other users in the organization through the Real-Time Hub.
Enhanced trigger options
Shipped (Q2 2024)
To represent the business conditions for monitoring, Data Activator will support more trigger conditions such as percentage changes over time, date/time functions, basic mathematic functions, signal absence detection and more.
Notebook integration
Shipped (Q1 2024)
Customers can natively query KQL databases from Microsoft Fabric notebooks. Use notebooks to run queries on KQL databases, save the resulting data frames, ingest data into KQL databases, and more.
Scale up to 100 MB/s for eventstream processing
Shipped (Q1 2024)
An eventstream can support events ingestion throughput up to 100 MB/s while ingesting data to KQL databases and lakehouses.
Customizable event data retention in eventstreams
Shipped (Q1 2024)
Customers have the flexibility to set the event data retention time, ensuring that event data is retained in their eventstreams according to their specific business requirements.
Enhancements to Event processor in Eventstream
Shipped (Q1 2024)
This includes three enhancements. First, personalize operation nodes and easily filter out 'null' values from your data. Second, manage and rename your column fields easily in the Aggregate operation. Third, Change your values to different data types using the Manage Fields operation .
Table optimization shortcut in eventstream lakehouse destinations
Shipped (Q1 2024)
The table optimization shortcut helps users by opening a notebook with a Spark job that compacts small streaming files in the lakehouse table.
Enhanced Custom App Details tab offers comprehensive endpoint information
Shipped (Q1 2024)
The Details tab of the Custom App source and destination offers comprehensive information about the eventstream endpoint. Available information includes connection strings in Kafka, Eventhub, and AMQP formats, as well as sample Java code for event ingestion and consumption using these three protocols.
Improved no-code stream processing designer
Shipped (Q1 2024)
Users have an improved eventstream editing experience, with more intuitive gestures in the no-code designer.
"Get data from Eventstream" in multiple Fabric items
Shipped (Q1 2024)
Customers are able to get event data from eventstreams in multiple Fabric items, including Lakehouse, KQL Database, and Reflex.
New eventstream destination to Data Activator
Shipped (Q1 2024)
The Data Activator destination option helps customers route events to Data Activator, where they can build triggers based on the event data from eventstreams natively.
Autoscale eventstreams
Shipped (Q1 2024)
This feature provides the flexibility to automatically scale capacity in response to event data traffic volume, enabling seamless improvement of eventstream throughput without disrupting your business operations.
Delta support to the KQL database OneLake representation
Shipped (Q1 2024)
Release Type: Public preview
KQL database data is already available in OneLake in parquet format. Adding Delta support enables other Fabric capabilities to use the OneLake presence as a table rather than as a list of files.
Administrative monitoring
Shipped (Q1 2024)
You can use administrative monitoring to search audit logs, and to track usage and adoption, capacity consumption, and availability.
Eventstream data preprocessing to KQL database destination
Shipped (Q1 2024)
The addition of the KQL database destination with data transformation enables customers to transform, enrich, and reduce the data volume before it's routed to KQL databases.
Improved Get Data experience
Shipped (Q1 2024)
Users experience a simplified and improved Get Data experience to get their data into a KQL database.
Cloud connection creation embedded in eventstreams
Shipped (Q1 2024)
Customers can create a cloud connection directly within their eventstreams, without navigating to another webpage.
Two ingestion modes for eventstream lakehouse destinations
Shipped (Q1 2024)
Eventstream lakehouse destinations support two ingestion modes: low latency and high throughput.