SqlGraphRegistrationContext¶
Creating Instance¶
SqlGraphRegistrationContext takes the following to be created:
SqlGraphRegistrationContext is created when:
PipelinesHandleris requested to handle DEFINE_SQL_GRAPH_ELEMENTS command (and defineSqlGraphElements)SqlGraphRegistrationContextis requested to process a SQL file
SqlGraphRegistrationContextState¶
When created, SqlGraphRegistrationContext creates a SqlGraphRegistrationContextState (with the defaultCatalog, the defaultDatabase and the defaultSqlConf).
Process SQL File¶
processSqlFile(
sqlText: String,
sqlFilePath: String,
spark: SparkSession): Unit
processSqlFile creates a SqlGraphRegistrationContext for this GraphRegistrationContext.
Warning
Why does processSqlFile creates a brand new SqlGraphRegistrationContext for the same GraphRegistrationContext it is executed with?!
processSqlFile splits the contents of the SQL file into separate queries and processes every SQL query.
processSqlFile is used when:
PipelinesHandleris requested to defineSqlGraphElements
Process Single SQL Query¶
processSqlQuery(
queryPlan: LogicalPlan,
queryOrigin: QueryOrigin): Unit
processSqlQuery handles (processes) the given LogicalPlan logical commands:
| Logical Command | Command Handler | Datasets |
|---|---|---|
| CreateFlowCommand | CreateFlowHandler | UnresolvedFlow (once disabled) |
| CreateMaterializedViewAsSelect | CreateMaterializedViewAsSelectHandler | Table (isStreamingTable disabled) UnresolvedFlow (once disabled) |
| CreateStreamingTable | CreateStreamingTableHandler | Table (isStreamingTable enabled) |
| CreateStreamingTableAsSelect | CreateStreamingTableAsSelectHandler | Table (isStreamingTable enabled) UnresolvedFlow (once disabled) |
| CreateView | CreatePersistedViewCommandHandler | PersistedView UnresolvedFlow (once disabled) |
| CreateViewCommand | CreateTemporaryViewHandler | TemporaryView UnresolvedFlow (once disabled) |
| SetCatalogCommand | SetCatalogCommandHandler | |
| SetCommand | SetCommandHandler | |
| SetNamespaceCommand | SetNamespaceCommandHandler |
splitSqlFileIntoQueries¶
splitSqlFileIntoQueries(
spark: SparkSession,
sqlFileText: String,
sqlFilePath: String): Seq[SqlQueryPlanWithOrigin]
splitSqlFileIntoQueries...FIXME
Logical Command Handlers¶
CreateFlowCommand¶
CreateFlowCommand logical commands are handled by CreateFlowHandler.
A flow name must be a single-part name (that is resolved against the current pipelines catalog and database).
The flowOperation of a CreateFlowCommand command must be InsertIntoStatement.
Warning
Only INSERT INTO ... BY NAME flows are supported in Spark Declarative Pipelines.
INSERT OVERWRITE flows are not supported.
IF NOT EXISTS not supported for flows.
Neither partition spec nor user-specified schema can be specified.
In the end, CreateFlowHandler requests this GraphRegistrationContext to register an UnresolvedFlow.
CreateMaterializedViewAsSelect¶
processSqlQuery handles CreateMaterializedViewAsSelect logical commands using CreateMaterializedViewAsSelectHandler.
CreateMaterializedViewAsSelectHandler requests this GraphRegistrationContext to register a table and a flow (that backs the materialized view).
CreateStreamingTable¶
processSqlQuery handles CreateStreamingTable logical commands using CreateStreamingTableHandler.
CreateStreamingTableHandler requests this SqlGraphRegistrationContextState to register a streaming table.
CreateStreamingTableAsSelect¶
processSqlQuery handles CreateStreamingTableAsSelect logical commands using CreateStreamingTableAsSelectHandler.
CreateStreamingTableAsSelectHandler requests this SqlGraphRegistrationContextState to register a streaming table and the accompanying flow (for the streaming table).
CreateView¶
processSqlQuery handles CreateView logical commands using CreatePersistedViewCommandHandler.
CreatePersistedViewCommandHandler requests this GraphRegistrationContext to register a PersistedView and the accompanying flow (for the PersistedView).
CreateViewCommand¶
processSqlQuery handles CreateViewCommand logical commands using CreateTemporaryViewHandler.
CreateTemporaryViewHandler requests this GraphRegistrationContext to register a TemporaryView and the accompanying flow (for the TemporaryView).
SetCatalogCommand¶
processSqlQuery handles SetCatalogCommand logical commands using SetCatalogCommandHandler.
SetCatalogCommandHandler requests this SqlGraphRegistrationContextState to setCurrentCatalog to the catalogName of the given SetCatalogCommand.
In the end, SetCatalogCommandHandler requests this SqlGraphRegistrationContextState to clearCurrentDatabase.
SetCommand¶
processSqlQuery handles SetCommand logical commands using SetCommandHandler.
SetCommandHandler requests this SqlGraphRegistrationContextState to setSqlConf with the key-value pair of the given SetCommand logical command.
RuntimeException
handle makes sure that the given SetCommand comes with a key = value pair or throws a RuntimeException:
Invalid SET command without key-value pair
Invalid SET command without value
SetNamespaceCommand¶
processSqlQuery handles SetNamespaceCommand logical commands using SetNamespaceCommandHandler.
SetNamespaceCommandHandler requests this SqlGraphRegistrationContextState for the following:
- For a
database-only, single-part namespace, setCurrentDatabase - For a
catalog.databasetwo-part namespace, setCurrentCatalog and setCurrentDatabase
SparkException
handle throws a SparkException for invalid namespaces:
Invalid schema identifier provided on USE command: [namespace]