FlowPlanner¶

FlowPlanner is used to plan a flow into a FlowExecution.

FlowPlanner is created alongside a GraphExecution (i.e., when PipelineExecution is requested to start a pipeline).

Creating Instance¶

FlowPlanner takes the following to be created:

• DataflowGraph
• PipelineUpdateContext
• Flow-to-Streaming-Trigger Conversion Function

FlowPlanner is created alongside a GraphExecution.

Flow-to-Streaming-Trigger Conversion Function¶

triggerFor: Flow => Trigger

FlowPlanner is given a function to convert a Flow into a streaming Trigger (Spark Structured Streaming) when created.

The triggerFor function is the streamTrigger function of the owning GraphExecution.

Plan DataflowGraph¶

plan(
  flow: ResolvedFlow): FlowExecution

plan creates a FlowExecution (for the given ResolvedFlow) as follows:

plan is used when:

• GraphExecution is requested to planAndStartFlow