FlowPlanner¶
FlowPlanner
is used to plan a flow into a FlowExecution.
FlowPlanner
is created alongside a GraphExecution (i.e., when PipelineExecution is requested to start a pipeline).
Creating Instance¶
FlowPlanner
takes the following to be created:
FlowPlanner
is created alongside a GraphExecution.
Flow-to-Streaming-Trigger Conversion Function¶
triggerFor: Flow => Trigger
FlowPlanner
is given a function to convert a Flow into a streaming Trigger
(Spark Structured Streaming) when created.
The triggerFor
function is the streamTrigger function of the owning GraphExecution.
Plan DataflowGraph¶
plan(
flow: ResolvedFlow): FlowExecution
plan
creates a FlowExecution (for the given ResolvedFlow) as follows:
FlowExecution | ResolvedFlow |
---|---|
BatchTableWrite | CompleteFlow |
StreamingTableWrite | StreamingFlow |
plan
is used when:
GraphExecution
is requested to planAndStartFlow