StreamingFlowExecution¶

StreamingFlowExecution is an extension of the FlowExecution abstraction for streaming flow executions that process data statefully using Spark Structured Streaming.

Contract (Subset)¶

Checkpoint Location¶

checkpointPath: String

Used when:

• StreamingTableWrite is requested to start a streaming query

Start Streaming Query¶

startStream(): StreamingQuery

See:

• SinkWrite
• StreamingTableWrite

Used when:

• StreamingFlowExecution is requested to executeInternal

Streaming Trigger¶

trigger: Trigger

Trigger (Structured Streaming)

See:

• SinkWrite
• StreamingTableWrite

Used when:

• FlowPlanner is requested to plan a StreamingFlow
• StreamingTableWrite is requested to execute the streaming query

Implementations¶

• SinkWrite
• StreamingTableWrite

executeInternal¶

executeInternal(): Future[Unit]

executeInternal prints out the following INFO message to the logs:

Starting [identifier] with checkpoint location [checkpointPath]

executeInternal starts the stream (with this SparkSession and sqlConf).

In the end, executeInternal awaits termination of the StreamingQuery.