Share via

SSIS Dataflow Engine

  • Article

Overview

The Dataflow engine is the software component in [[articles:SQL Server Integration Services|SSIS]] that powers the [[articles:SSIS Dataflow Task|Dataflow Task]]. It moves data from one place to another in memory-resident data structures called buffers (sometimes called pipeline buffers). A buffer is effectively a subset of all the data that is being processed by the Dataflow engine.

The use of buffers allows the Dataflow engine to process more data than can fit into the memory space of the computer at any one time.  If the amount of memory available to the Dataflow engine at a point-in-time exceeds the amount required to process all the data then buffers will be spooled to disk.

Synonyms

You may see the Dataflow engine referred to as "the pipeline engine" or more simply "the pipeline". The Dataflow engine is the underlying engine that powers the [[articles:SSIS Dataflow Task|Dataflow Task]] and the two can be considered synonymous with one another.