Skip to content

FlowPlanner

FlowPlanner is used to plan a flow into a FlowExecution.

FlowPlanner is created alongside a GraphExecution (i.e., when PipelineExecution is requested to start a pipeline).

Creating Instance

FlowPlanner takes the following to be created:

FlowPlanner is created alongside a GraphExecution.

Flow-to-Streaming-Trigger Conversion Function

triggerFor: Flow => Trigger

FlowPlanner is given a function to convert a Flow into a streaming Trigger (Spark Structured Streaming) when created.

The triggerFor function is the streamTrigger function of the owning GraphExecution.

Plan DataflowGraph

plan(
  flow: ResolvedFlow): FlowExecution

plan creates a FlowExecution (for the given ResolvedFlow) as follows:

FlowExecution ResolvedFlow
BatchTableWrite CompleteFlow
StreamingTableWrite StreamingFlow

plan is used when: