DataStreamWriter Class

Definition

Namespace:

Microsoft.Spark.Sql.Streaming

Assembly:

Microsoft.Spark.dll

Package:

Microsoft.Spark v1.0.0

DataStreamWriter provides functionality to write a streaming DataFrame to external storage systems (e.g. file systems, key-value stores, etc).

public sealed class DataStreamWriter

type DataStreamWriter = class

Public NotInheritable Class DataStreamWriter

Inheritance

Object

DataStreamWriter

Methods

Foreach(IForeachWriter)	Sets the output of the streaming query to be processed using the provided writer object. See IForeachWriter for more details on the lifecycle and semantics.
ForeachBatch(Action<DataFrame,Int64>)	Sets the output of the streaming query to be processed using the provided function. This is supported only in the micro-batch execution modes (that is, when the trigger is not continuous). In every micro-batch, the provided function will be called in every micro-batch with (i) the output rows as a DataFrame and (ii) the batch identifier. The batchId can be used to deduplicate and transactionally write the output (that is, the provided Dataset) to external systems. The output DataFrame is guaranteed to exactly same for the same batchId (assuming all operations are deterministic in the query).
Format(String)	Specifies the underlying output data source.
Option(String, Boolean)	Adds an output option for the underlying data source.
Option(String, Double)	Adds an output option for the underlying data source.
Option(String, Int64)	Adds an output option for the underlying data source.
Option(String, String)	Adds an output option for the underlying data source.
Options(Dictionary<String,String>)	Adds output options for the underlying data source.
OutputMode(OutputMode)	Specifies how data of a streaming DataFrame is written to a streaming sink.
OutputMode(String)	Specifies how data of a streaming DataFrame is written to a streaming sink.
PartitionBy(String[])	Partitions the output by the given columns on the file system. If specified, the output is laid out on the file system similar to Hive's partitioning scheme.
QueryName(String)	Specifies the name of the StreamingQuery that can be started with start(). This name must be unique among all the currently active queries in the associated SQLContext.
Start(String)	Starts the execution of the streaming query.
Trigger(Trigger)	Sets the trigger for the stream query.