TriggeredGraphExecution¶
TriggeredGraphExecution is a GraphExecution for the DataflowGraph and PipelineUpdateContext.
Creating Instance¶
TriggeredGraphExecution takes the following to be created:
- DataflowGraph
- PipelineUpdateContext
-
onCompletionCallback (RunTerminationReason => Unit, default: do nothing) -
Clock(default:SystemClock)
TriggeredGraphExecution is created when:
PipelineExecutionis requested to run a pipeline update
Topological Execution Thread¶
topologicalExecutionThread: Option[Thread]
topologicalExecutionThread is the Topological Execution thread of execution of this pipeline update.
topologicalExecutionThread is initialized and started when TriggeredGraphExecution is requested to start.
topologicalExecutionThread runs until awaitCompletion or stopInternal.
Start Pipeline¶
start registers the stream listener.
start requests this PipelineUpdateContext for the flows to be refreshed and...FIXME
start creates a Topological Execution thread and starts its execution.
Create Topological Execution Thread¶
buildTopologicalExecutionThread(): Thread
buildTopologicalExecutionThread creates a new thread of execution known as Topological Execution.
When started, the thread does topological execution.
topologicalExecution¶
topologicalExecution(): Unit
topologicalExecution finds the flows in QUEUED and RUNNING states or failed but can be re-tried.
For each flow in RUNNING state, topologicalExecution...FIXME
topologicalExecution checks leaking permits.
FIXME Explain
topologicalExecution starts flows that are ready to start.
Start Single Flow¶
startFlow(
flow: ResolvedFlow): Unit
startFlow prints out the following INFO message to the logs:
Starting flow [flow_identifier]
startFlow requests this PipelineUpdateContext for the FlowProgressEventLogger to recordPlanningForBatchFlow.
startFlow planAndStartFlow.
startFlow...FIXME
Streaming Trigger¶
GraphExecution
streamTrigger(
flow: Flow): Trigger
streamTrigger is part of the GraphExecution abstraction.
streamTrigger is AvailableNowTrigger (Spark Structured Streaming).
awaitCompletion¶
awaitCompletion waits for this Topological Execution thread to die.
stopInternal¶
stopInternal(
stopTopologicalExecutionThread: Boolean): Unit
stopInternal...FIXME
stopInternal is used when:
Stop Execution¶
stop stopInternal (with stopTopologicalExecutionThread flag enabled).