UnresolvedFlow¶

UnresolvedFlow is a Flow that represents a flow in the Python and SQL transformations in Spark Declarative Pipelines:

• register_flow in PySpark's decorators
• CREATE FLOW ... AS INSERT INTO ... BY NAME
• CREATE MATERIALIZED VIEW
• CREATE STREAMING TABLE ... AS
• CREATE VIEW and the other variants of CREATE VIEW

UnresolvedFlow is registered to a GraphRegistrationContext with register a flow.

UnresolvedFlow is analyzed and resolved to ResolvedFlow (by FlowResolver when DataflowGraph is requested to resolve).

UnresolvedFlow must have unique identifiers (or an AnalysisException is reported).

Creating Instance¶

UnresolvedFlow takes the following to be created:

• TableIdentifier
• Flow destination (TableIdentifier)
• FlowFunction
• QueryContext
• SQL Config
• once flag
• QueryOrigin

UnresolvedFlow is created when:

• PipelinesHandler is requested to define a flow
• SqlGraphRegistrationContext is requested to handle the following logical commands:
  • CreateFlowCommand
  • CreateMaterializedViewAsSelect
  • CreateView
  • CreateStreamingTableAsSelect
  • CreateViewCommand

once Flag¶

UnresolvedFlow is given the once flag when created.

once flag is disabled (false) explicitly for the following:

• CreateFlowHandler
• CreateMaterializedViewAsSelectHandler
• CreatePersistedViewCommandHandler
• CreateStreamingTableAsSelectHandler
• CreateTemporaryViewHandler
• PipelinesHandler is requested to define a flow