StreamingFlowExecution¶

StreamingFlowExecution is an extension of the FlowExecution abstraction for streaming flow executions that process data statefully using Spark Structured Streaming.

Contract¶

Execute Streaming Query¶

startStream(): StreamingQuery

See:

• StreamingTableWrite

Used when:

• StreamingFlowExecution is requested to executeInternal

Streaming Trigger¶

trigger: Trigger

See:

• StreamingTableWrite

Used when:

• FlowPlanner is requested to plan a StreamingFlow
• StreamingTableWrite is requested to execute the streaming query

Implementations¶

• StreamingTableWrite

executeInternal¶

executeInternal(): Future[Unit]

executeInternal prints out the following INFO message to the logs:

Starting [identifier] with checkpoint location [checkpointPath]"

executeInternal starts the stream (with this SparkSession and sqlConf).

In the end, executeInternal awaits termination of the StreamingQuery.