OutputMode of a streaming query described what data is written to a streaming sink.
OutputMode is specified using DataStreamWriter.outputMode.
append) is the default output mode that writes "new" rows only.
In streaming aggregations, a "new" row is when the intermediate state becomes final, i.e. when new events for the grouping key can only be considered late which is when watermark moves past the event time of the key.
Append output mode requires that a streaming query defines event-time watermark (using withWatermark operator) on the event time column that is used in aggregation (directly or using window standard function).
Required for datasets with
FileFormat format (to create FileStreamSink)
Append is mandatory when multiple
flatMapGroupsWithState operators are used in a structured query.
complete) writes all the rows of a Result Table (and corresponds to a traditional batch structured query).
Complete mode does not drop old aggregation state and preserves all data in the Result Table.
update) writes only the rows that were updated (every time there are updates).