StreamingTableWrite¶
StreamingTableWrite
is a StreamingFlowExecution.
When executed, StreamingTableWrite
starts a streaming query to append new rows to an output table.
Creating Instance¶
StreamingTableWrite
takes the following to be created:
- TableIdentifier
- ResolvedFlow
- DataflowGraph
- PipelineUpdateContext
- Checkpoint Path
- Streaming Trigger
- Output table
- SQL Configuration
StreamingTableWrite
is created when:
FlowPlanner
is requested to plan a StreamingFlow
Execute Streaming Query¶
StreamingFlowExecution
startStream(): StreamingQuery
startStream
is part of the StreamingFlowExecution abstraction.
startStream
builds the logical query plan of this flow's structured query (requesting the DataflowGraph to reanalyze this flow).
startStream
creates a DataStreamWriter
(Spark Structured Streaming) with the following:
DataStreamWriter 's Property | Value |
---|---|
queryName | This displayName |
checkpointLocation option | This checkpoint path |
trigger | This streaming trigger |
outputMode | Append (always) |
format | The format of this output table (only when defined) |
In the end, startStream
starts the streaming write query to this output table.