SqlGraphRegistrationContext¶
Creating Instance¶
SqlGraphRegistrationContext takes the following to be created:
SqlGraphRegistrationContext is created when:
PipelinesHandleris requested to handle DEFINE_SQL_GRAPH_ELEMENTS command (and defineSqlGraphElements)SqlGraphRegistrationContextis requested to processSqlFile
Process SQL Definition File¶
processSqlFile(
sqlText: String,
sqlFilePath: String,
spark: SparkSession): Unit
processSqlFile creates a SqlGraphRegistrationContext for this GraphRegistrationContext.
Warning
Why does processSqlFile creates a brand new SqlGraphRegistrationContext for the same GraphRegistrationContext it is executed with?!
processSqlFile splits the contents of the SQL file into separate queries and processes every SQL query.
processSqlFile is used when:
PipelinesHandleris requested to defineSqlGraphElements
Process Single SQL Query¶
processSqlQuery(
queryPlan: LogicalPlan,
queryOrigin: QueryOrigin): Unit
processSqlQuery handles (processes) the given LogicalPlan logical commands:
SetCommandSetNamespaceCommandSetCatalogCommandCreateViewCreateViewCommand- CreateMaterializedViewAsSelect
CreateStreamingTableAsSelectCreateStreamingTable- CreateFlowCommand
splitSqlFileIntoQueries¶
splitSqlFileIntoQueries(
spark: SparkSession,
sqlFileText: String,
sqlFilePath: String): Seq[SqlQueryPlanWithOrigin]
splitSqlFileIntoQueries...FIXME
CreateFlowCommand¶
CreateFlowCommand logical commands are handled by CreateFlowHandler.
A flow name must be a single-part name (that is resolved against the current pipelines catalog and database).
The flowOperation of a CreateFlowCommand command must be InsertIntoStatement.
Note
Only INSERT INTO ... BY NAME flows are supported in Spark Declarative Pipelines.
INSERT OVERWRITE flows are not supported.
IF NOT EXISTS not supported for flows.
Neither partition spec nor user-specified schema can be specified.
In the end, CreateFlowHandler requests this GraphRegistrationContext to register an UnresolvedFlow.
CreateMaterializedViewAsSelect¶
CreateMaterializedViewAsSelect logical commands are handled by CreateMaterializedViewAsSelectHandler.
CreateMaterializedViewAsSelectHandler requests this GraphRegistrationContext to register a table and a flow (that backs the materialized view).