TriggeredGraphExecution¶

TriggeredGraphExecution is a GraphExecution for the DataflowGraph and PipelineUpdateContext.

Creating Instance¶

TriggeredGraphExecution takes the following to be created:

• DataflowGraph
• PipelineUpdateContext
• onCompletion Callback (RunTerminationReason => Unit, default: do nothing)
• Clock (default: SystemClock)

TriggeredGraphExecution is created when:

• PipelineExecution is requested to run a pipeline update

Topological Execution Thread¶

topologicalExecutionThread: Option[Thread]

topologicalExecutionThread is the Topological Execution thread of execution of this pipeline update.

topologicalExecutionThread is initialized and started when TriggeredGraphExecution is requested to start.

topologicalExecutionThread runs until awaitCompletion or stopInternal.

Start Pipeline¶

start(): Unit

start registers the stream listener.

start requests this PipelineUpdateContext for the flows to be refreshed and...FIXME

start creates a Topological Execution thread and starts its execution.

Create Topological Execution Thread¶

buildTopologicalExecutionThread(): Thread

buildTopologicalExecutionThread creates a new thread of execution known as Topological Execution.

When started, the thread does topological execution.

topologicalExecution¶

topologicalExecution(): Unit

topologicalExecution finds the flows in QUEUED and RUNNING states or failed but can be re-tried.

For each flow in RUNNING state, topologicalExecution...FIXME

topologicalExecution checks leaking permits.

topologicalExecution starts flows that are ready to start.

Start Single Flow¶

startFlow(
  flow: ResolvedFlow): Unit

startFlow prints out the following INFO message to the logs:

Starting flow [flow_identifier]

startFlow requests this PipelineUpdateContext for the FlowProgressEventLogger to recordPlanningForBatchFlow.

startFlow planAndStartFlow.

startFlow...FIXME

Streaming Trigger¶

streamTrigger(
  flow: Flow): Trigger

streamTrigger is AvailableNowTrigger (Spark Structured Streaming).

awaitCompletion¶

awaitCompletion(): Unit

awaitCompletion waits for this Topological Execution thread to die.

stopInternal¶

stopInternal(
  stopTopologicalExecutionThread: Boolean): Unit

stopInternal...FIXME

stopInternal is used when:

• TriggeredGraphExecution is requested to start and stop

Stop Execution¶

stop(): Unit

stop stopInternal (with stopTopologicalExecutionThread flag enabled).