GraphRegistrationContext¶
GraphRegistrationContext
is a registry of tables, views, and flows in a pipeline (dataflow graph).
GraphRegistrationContext
is required to create a new SqlGraphRegistrationContext.
Eventually, GraphRegistrationContext
becomes a DataflowGraph (to create a PipelineUpdateContextImpl to run a pipeline).
Creating Instance¶
GraphRegistrationContext
takes the following to be created:
- Default Catalog
- Default Database
- Default SQL Configuration Properties
GraphRegistrationContext
is created when:
DataflowGraphRegistry
is requested to createDataflowGraph
Create DataflowGraph¶
toDataflowGraph: DataflowGraph
toDataflowGraph
creates a new DataflowGraph with the tables, views, and flows fully-qualified, resolved, and de-duplicated.
AnalysisException
toDataflowGraph
reports an AnalysisException
for a GraphRegistrationContext
with no tables and no PersistedView
s (in the views registry).
toDataflowGraph
is used when:
PipelinesHandler
is requested to start a pipeline run
assertNoDuplicates¶
assertNoDuplicates(
qualifiedTables: Seq[Table],
validatedViews: Seq[View],
qualifiedFlows: Seq[UnresolvedFlow]): Unit
assertNoDuplicates
...FIXME
assertFlowIdentifierIsUnique¶
assertFlowIdentifierIsUnique(
flow: UnresolvedFlow,
datasetType: DatasetType,
flows: Seq[UnresolvedFlow]): Unit
assertFlowIdentifierIsUnique
throws an AnalysisException
if the given UnresolvedFlow's identifier is used by multiple flows (among the given flows
):
Flow [flow_name] was found in multiple datasets: [dataset_names]
Tables¶
GraphRegistrationContext
creates an empty registry of Tables when created.
A new Table is added when registerTable.
Views¶
GraphRegistrationContext
creates an empty registry of Views when created.
Flows¶
GraphRegistrationContext
creates an empty registry of UnresolvedFlows when created.
Register Table¶
registerTable(
tableDef: Table): Unit
registerTable
adds the given Table to the tables registry.
registerTable
is used when:
PipelinesHandler
is requested to define a dataset
Register Flow¶
registerFlow(
flowDef: UnresolvedFlow): Unit
registerFlow
adds the given UnresolvedFlow to the flows registry.
registerFlow
is used when:
PipelinesHandler
is requested to define a flowSqlGraphRegistrationContext
is requested to handle the following logical commands: