StreamingFlowExecution¶

StreamingFlowExecution is an extension of the FlowExecution abstraction for streaming flow executions that process data statefully using Spark Structured Streaming.

Contract (Subset)¶

Checkpoint Location¶

checkpointPath: String

Path for the most recent checkpoint of this streaming flow execution

See:

• SinkWrite
• StreamingTableWrite

Used when:

• StreamingTableWrite is requested to start a streaming query

Resolved Flow¶

flow: ResolvedFlow

See:

• SinkWrite
• StreamingTableWrite

Start Streaming Query¶

startStream(): StreamingQuery

See:

• SinkWrite
• StreamingTableWrite

Used when:

• StreamingFlowExecution is requested to executeInternal

Start Streaming Query¶

startStream(): StreamingQuery

See:

• SinkWrite
• StreamingTableWrite

Used when:

• StreamingFlowExecution is requested to executeInternal

sqlConf¶

sqlConf: Map[String, String]

Spark configuration properties that must be set upon starting this flow

See:

• SinkWrite
• StreamingTableWrite

Used when:

• StreamingFlowExecution is requested to executeInternal

Streaming Trigger¶

trigger: Trigger

Trigger (Spark Structured Streaming)

See:

• SinkWrite
• StreamingTableWrite

Used when:

• SinkWrite is requested to startStream
• StreamingTableWrite is requested to startStream

Implementations¶

• SinkWrite
• StreamingTableWrite

executeInternal¶

executeInternal(): Future[Unit]

executeInternal prints out the following INFO message to the logs:

Starting [identifier] with checkpoint location [checkpointPath]

executeInternal starts the stream (with this SparkSession and sqlConf).

In the end, executeInternal awaits termination of the StreamingQuery.