SqlGraphRegistrationContext¶

Creating Instance¶

SqlGraphRegistrationContext takes the following to be created:

GraphRegistrationContext

SqlGraphRegistrationContext is created when:

PipelinesHandler is requested to handle DEFINE_SQL_GRAPH_ELEMENTS command (and defineSqlGraphElements)
SqlGraphRegistrationContext is requested to process a SQL file

SqlGraphRegistrationContextState¶

When created, SqlGraphRegistrationContext creates a SqlGraphRegistrationContextState (with the defaultCatalog, the defaultDatabase and the defaultSqlConf).

Process SQL File¶

processSqlFile(
  sqlText: String,
  sqlFilePath: String,
  spark: SparkSession): Unit

processSqlFile creates a SqlGraphRegistrationContext for this GraphRegistrationContext.

Warning

Why does processSqlFile creates a brand new SqlGraphRegistrationContext for the same GraphRegistrationContext it is executed with?!

processSqlFile splits the contents of the SQL file into separate queries and processes every SQL query.

processSqlFile is used when:

PipelinesHandler is requested to defineSqlGraphElements

Process Single Logical Command¶

processSqlQuery(
  queryPlan: LogicalPlan,
  queryOrigin: QueryOrigin): Unit

processSqlQuery handles (processes) the given LogicalPlan logical commands:

Logical Command	Command Handler	Datasets
CreateFlowCommand	CreateFlowHandler	UnresolvedFlow (once disabled)
CreateMaterializedViewAsSelect	CreateMaterializedViewAsSelectHandler	Table (isStreamingTable disabled) UnresolvedFlow (once disabled)
CreateStreamingTable	CreateStreamingTableHandler	Table (isStreamingTable enabled)
CreateStreamingTableAsSelect	CreateStreamingTableAsSelectHandler	Table (isStreamingTable enabled) UnresolvedFlow (once disabled)
CreateView	CreatePersistedViewCommandHandler	PersistedView UnresolvedFlow (once disabled)
CreateViewCommand	CreateTemporaryViewHandler	TemporaryView UnresolvedFlow (once disabled)
SetCatalogCommand	SetCatalogCommandHandler
SetCommand	SetCommandHandler
SetNamespaceCommand	SetNamespaceCommandHandler

splitSqlFileIntoQueries¶

splitSqlFileIntoQueries(
  spark: SparkSession,
  sqlFileText: String,
  sqlFilePath: String): Seq[SqlQueryPlanWithOrigin]

splitSqlFileIntoQueries...FIXME

Logical Command Handlers¶

CreateFlowCommand¶

CreateFlowCommand logical commands are handled by CreateFlowHandler.

A flow name must be a single-part name (that is resolved against the current pipelines catalog and database).

The flowOperation of a CreateFlowCommand command must be InsertIntoStatement.

Warning

Only INSERT INTO ... BY NAME flows are supported in Spark Declarative Pipelines.

INSERT OVERWRITE flows are not supported.

IF NOT EXISTS not supported for flows.

Neither partition spec nor user-specified schema can be specified.

In the end, CreateFlowHandler requests this GraphRegistrationContext to register an UnresolvedFlow.

CreateMaterializedViewAsSelect¶

processSqlQuery handles CreateMaterializedViewAsSelect logical commands using CreateMaterializedViewAsSelectHandler.

CreateMaterializedViewAsSelectHandler requests this GraphRegistrationContext to register a table and a flow (that backs the materialized view).

CreateStreamingTable¶

processSqlQuery handles CreateStreamingTable logical commands using CreateStreamingTableHandler.

CreateStreamingTableHandler requests this SqlGraphRegistrationContextState to register a streaming table.

CreateStreamingTableAsSelect¶

processSqlQuery handles CreateStreamingTableAsSelect logical commands using CreateStreamingTableAsSelectHandler.

CreateStreamingTableAsSelectHandler requests this SqlGraphRegistrationContextState to register a streaming table and the accompanying flow (for the streaming table).

CreateView¶

processSqlQuery handles CreateView logical commands using CreatePersistedViewCommandHandler.

CreatePersistedViewCommandHandler requests this GraphRegistrationContext to register a PersistedView and the accompanying flow (for the PersistedView).

CreateViewCommand¶

processSqlQuery handles CreateViewCommand logical commands using CreateTemporaryViewHandler.

CreateTemporaryViewHandler requests this GraphRegistrationContext to register a TemporaryView and the accompanying flow (for the TemporaryView).

SetCatalogCommand¶

processSqlQuery handles SetCatalogCommand logical commands using SetCatalogCommandHandler.

SetCatalogCommandHandler requests this SqlGraphRegistrationContextState to setCurrentCatalog to the catalogName of the given SetCatalogCommand.

In the end, SetCatalogCommandHandler requests this SqlGraphRegistrationContextState to clearCurrentDatabase.

SetCommand¶

processSqlQuery handles SetCommand logical commands using SetCommandHandler.

SetCommandHandler requests this SqlGraphRegistrationContextState to setSqlConf with the key-value pair of the given SetCommand logical command.

RuntimeException

handle makes sure that the given SetCommand comes with a key = value pair or throws a RuntimeException:

Invalid SET command without key-value pair

Invalid SET command without value

SetNamespaceCommand¶

processSqlQuery handles SetNamespaceCommand logical commands using SetNamespaceCommandHandler.

SetNamespaceCommandHandler requests this SqlGraphRegistrationContextState for the following:

For a database-only, single-part namespace, setCurrentDatabase
For a catalog.database two-part namespace, setCurrentCatalog and setCurrentDatabase

SparkException

handle throws a SparkException for invalid namespaces:

Invalid schema identifier provided on USE command: [namespace]