PipelineUpdateContext¶

PipelineUpdateContext is an abstraction of pipeline update contexts that can refreshTables (among other things).

Contract (Subset)¶

fullRefreshTables¶

fullRefreshTables: TableFilter

TableFilter of the tables to be fully refreshed in a pipeline update run

See:

• PipelineUpdateContextImpl

Used when:

• DatasetManager is requested to constructFullRefreshSet
• PipelineExecution is requested to start a pipeline update
• PipelineUpdateContext is requested to refreshFlows
• State is requested to find the inputs to reset (state of)

refreshTables Table Filter¶

refreshTables: TableFilter

Used when:

• DatasetManager is requested to constructFullRefreshSet
• PipelineUpdateContext is requested to refreshFlows

Storage Root¶

storageRoot: String

The root storage location of pipeline metadata (e.g., checkpoints of streaming flows)

Used when:

• FlowSystemMetadata is requested to flowCheckpointsDirOpt

Unresolved Dataflow Graph¶

unresolvedGraph: DataflowGraph

The unresolved DataflowGraph of this pipeline update (pipeline run) to execute

Used when:

• PipelineExecution is requested to resolve this unresolved DataflowGraph

Implementations¶

• PipelineUpdateContextImpl

PipelineExecution¶

pipelineExecution: PipelineExecution

PipelineUpdateContext creates a PipelineExecution when created.

The PipelineExecution is created for this PipelineUpdateContext.

refreshFlows¶

refreshFlows: FlowFilter

refreshFlows...FIXME

refreshFlows is used when:

• TriggeredGraphExecution is requested to start

Initialize Dataflow Graph¶

initializeGraph(): DataflowGraph

initializeGraph...FIXME

initializeGraph is used when:

• PipelineExecution is requested to start the pipeline