OutputMode¶

OutputMode of a streaming query described what data is written to a streaming sink.

OutputMode is specified using DataStreamWriter.outputMode.

Output Modes¶

Append¶

Append (alias: append) is the default output mode that writes "new" rows only.

In streaming aggregations, a "new" row is when the intermediate state becomes final, i.e. when new events for the grouping key can only be considered late which is when watermark moves past the event time of the key.

Append output mode requires that a streaming query defines event-time watermark (using withWatermark operator) on the event time column that is used in aggregation (directly or using window standard function).

Required for datasets with FileFormat format (to create FileStreamSink)

Append is mandatory when multiple flatMapGroupsWithState operators are used in a structured query.

Complete¶

Complete (alias: complete) writes all the rows of a Result Table (and corresponds to a traditional batch structured query).

Complete mode does not drop old aggregation state and preserves all data in the Result Table.

Supported only for streaming aggregations (as asserted by UnsupportedOperationChecker).

Update¶

Update (alias: update) writes only the rows that were updated (every time there are updates).

For queries that are not streaming aggregations, Update is equivalent to the Append output mode.