StreamingTableWrite¶
StreamingTableWrite is a StreamingFlowExecution.
When executed, StreamingTableWrite starts a streaming query to append new rows to an output table.
Creating Instance¶
StreamingTableWrite takes the following to be created:
- TableIdentifier
- ResolvedFlow
- DataflowGraph
- PipelineUpdateContext
- Checkpoint Path
- Streaming Trigger
- Output table
- SQL Configuration
StreamingTableWrite is created when:
FlowPlanneris requested to plan a StreamingFlow
Execute Streaming Query¶
StreamingFlowExecution
startStream(): StreamingQuery
startStream is part of the StreamingFlowExecution abstraction.
startStream builds the logical query plan of this flow's structured query (requesting the DataflowGraph to reanalyze this flow).
startStream creates a DataStreamWriter (Spark Structured Streaming) with the following:
DataStreamWriter's Property | Value |
|---|---|
queryName | This displayName |
checkpointLocation option | This checkpoint path |
trigger | This streaming trigger |
outputMode | Append (always) |
format | The format of this output table (only when defined) |
In the end, startStream starts the streaming write query to this output table.