DataflowGraph¶

Creating Instance¶

DataflowGraph takes the following to be created:

• Flows
• Tables
• Views

DataflowGraph is created when:

• DataflowGraph is requested to reanalyzeFlow
• GraphRegistrationContext is requested to convert to a DataflowGraph

reanalyzeFlow¶

reanalyzeFlow(
  srcFlow: Flow): ResolvedFlow

reanalyzeFlow finds the upstream datasets.

reanalyzeFlow finds the upstream flows (for the upstream datasets that could be found in the resolvedFlows registry).

reanalyzeFlow finds the upstream views (for the upstream datasets that could be found in the view registry).

reanalyzeFlow creates a new (sub)DataflowGraph for the upstream flows, views and a single table (the destination of the given Flow).

reanalyzeFlow requests the subgraph to resolve and returns the ResolvedFlow for the given Flow.

reanalyzeFlow is used when:

• BatchTableWrite is requested to executeAsync (and executeInternal)
• StreamingTableWrite is requested to executeAsync (and startStream)

Resolve¶

resolve(): DataflowGraph

resolve...FIXME

resolve is used when:

• DataflowGraph is requested to reanalyzeFlow
• PipelineExecution is requested to initializeGraph

Validate¶

validate(): DataflowGraph

validate...FIXME

validate is used when:

• PipelineExecution is requested to initialize the dataflow graph