PipelineUpdateContext¶
PipelineUpdateContext is an abstraction of pipeline update contexts that can refreshTables (among other things).
Contract (Subset)¶
refreshTables Table Filter¶
refreshTables: TableFilter
Used when:
DatasetManageris requested to constructFullRefreshSetPipelineUpdateContextis requested to refreshFlows
Root Storage Location¶
storageRoot: String
The root storage location of pipeline metadata (e.g., checkpoints for streaming flows)
Used when:
FlowSystemMetadatais requested to flowCheckpointsDirOpt
Unresolved Dataflow Graph¶
unresolvedGraph: DataflowGraph
The unresolved DataflowGraph of this pipeline update (pipeline run) to execute
Used when:
PipelineExecutionis requested to resolve this unresolved DataflowGraph
Implementations¶
PipelineExecution¶
pipelineExecution: PipelineExecution
PipelineUpdateContext creates a PipelineExecution when created.
The PipelineExecution is created for this PipelineUpdateContext.
refreshFlows¶
refreshFlows: FlowFilter
Final Method
refreshFlows is a Scala final method and may not be overridden in subclasses.
Learn more in the Scala Language Specification.
refreshFlows...FIXME
refreshFlows is used when:
TriggeredGraphExecutionis requested to start
Initialize Dataflow Graph¶
initializeGraph(): DataflowGraph
initializeGraph...FIXME
initializeGraph is used when:
PipelineExecutionis requested to start the pipeline