Share via

SQL Server and Microsoft StreamInsight

  • Article

As you might recall from our blog post on April 22nd, StreamInsight is packaged and licensed through Microsoft SQL Server. However, customers have pointed out various different deployment scenarios since our first StreamInsight CTPs. We reflected those in our approach to licensing, packaging and deployment of the product as follows:

  • We added Microsoft StreamInsight to the SQL Server flight of products to provide our data platform customers and partners with all the necessary components in one convenient package.
  • We decided to make StreamInsight installation and deployment as lightweight and flexible as possible. For instance, StreamInsight does not strictly require you to install any other SQL Server component besides StreamInsight.
  • Our adapter framework allows you to connect to other SQL Server and Microsoft platform products as data sources and sinks based on your scenario and your needs for deeper integration.

The rationale behind this is that we see event-driven applications emerge as a new paradigm of data-driven applications. For those scenarios, we view event processing as data processing and Microsoft's data platform would be the natural home for Microsoft's platform for event processing. However, customers have also pointed us to various use cases and scenarios that go beyond what is typically covered by Microsoft’s data platform.

Let's look at a couple of examples. Think, for instance, of static reference data that is not part of the incoming stream that StreamInsight processes. Instead, it is kept and maintained in a separate metadata store. You might still find it useful to correlate some of the incoming events from the stream with the corresponding reference data, for instance to drive grouping over a field like ZIP code that is only available from the reference data.
 
In many operational scenarios, for example in manufacturing, you have to deal with high input data rates. What is interesting in those scenarios are aggregates of the data, how the data changes, and the patterns or curves that the event data follows over time. Here, approaches do not scale that require you to put the data at rest in the database first and then run the same report over it again and again to check for the interesting patterns. Instead, you want to analyze the data while it is in flight, identify the patterns or complex events relevant to your business and then act on those.
 
Technically, StreamInsight facilitates these various deployment models by doing all its query processing in memory. StreamInsight has its own in-memory engine or runtime that does not need to write event data to disk.  Therefore, StreamInsight does not require you to install other SQL Server products if your scenario is already covered by the StreamInsight engine capabilities. Running StreamInsight just requires the .NET framework. This leads us to a follow-up question about the relationship between StreamInsight's engine and the CLR runtime. Although StreamInsight looks and feels like .NET, the StreamInsight engine does not depend on the garbage collector from the .NET runtime for its internal memory management. It instead uses its own native memory manager for the events to provide high throughput and predictable low latency.
 
SQL Server with StreamInsight provides you with the flexible platform to address all the various deployment scenarios mentioned above: You can either use existing or write your own input and output adapters to SQL Server engine databases, to Analysis Services cubes, to Velocity caches, BizTalk, or SQL Server Parallel Data Warehouse if you need to keep state or access persisted data. Or you can simply inject StreamInsight into your event data pipeline in order to do all the processing, transformations, calculations and analytics while the data is in flight. The most exciting deployments are those where you mix-and-match and connect all the pieces to a rich end-to-end application.

Regards,
Torsten Grabs

Comments

  • Anonymous
    June 08, 2010
    The comment has been removed

  • Anonymous
    January 12, 2011
    I'd agree with Eniola, I investigated some CEP solutions a year or so ago, but the project got mothballed. Coming back to it, I was quite excited to see StreamInsight, but then was almost immediately disappointed by the fact it's only available for the enterprise SQL installations. Looking at other options now. This may be a good product, but it's a bad rollout MS.

  • Anonymous
    January 13, 2011
    The comment has been removed