DeltaAnalysis Logical Resolution Rule¶

DeltaAnalysis is a logical resolution rule (Spark SQL).

Creating Instance¶

DeltaAnalysis takes the following to be created:

• SparkSession (Spark SQL)

DeltaAnalysis is created when:

• DeltaSparkSessionExtension is requested to register Delta Lake-specific session extensions

Executing Rule¶

apply(
  plan: LogicalPlan): LogicalPlan

apply resolves the following logical operators.

AppendData¶

For AppendData (Spark SQL), apply tries to destructure it to a pair of DataSourceV2Relation (Spark SQL) and a DeltaTableV2.

apply proceeds for AppendData operators that are not isByName.

apply...FIXME

CloneTableStatement¶

resolveCloneCommand¶

resolveCloneCommand(
  targetPlan: LogicalPlan,
  sourceTbl: CloneSource,
  statement: CloneTableStatement): LogicalPlan

resolveCloneCommand determines the SaveMode (in the following order):

• Overwrite for isReplaceCommand
• Ignore for ifNotExists
• ErrorIfExists for all other cases

In the end, resolveCloneCommand creates a CreateDeltaTableCommand logical operator.

CreateTableLikeCommand¶

DataSourceV2Relation¶

DeleteFromTable¶

DeltaMergeInto¶

DeltaReorgTable¶

DeltaTable¶

DeltaTableValueFunction¶

MergeIntoTable¶

apply resolves MergeIntoTable (Spark SQL) logical command into a DeltaMergeInto.

apply creates the following for the WHEN MATCHED actions:

• DeltaMergeIntoMatchedDeleteClauses for DeleteActions
• DeltaMergeIntoMatchedUpdateClauses for UpdateActions and UpdateStarActions

apply throws an AnalysisException for other actions:

[name] clauses cannot be part of the WHEN MATCHED clause in MERGE INTO.

apply creates the following for the WHEN NOT MATCHED actions:

• DeltaMergeIntoNotMatchedInsertClauses for InsertActions and InsertStarActions

apply throws an AnalysisException for other actions:

[name] clauses cannot be part of the WHEN NOT MATCHED clause in MERGE INTO.

apply creates the following for the WHEN NOT MATCHED BY SOURCE clauses:

• DeltaMergeIntoNotMatchedBySourceDeleteClauses for DeleteActions
• DeltaMergeIntoNotMatchedBySourceUpdateClauses for UpdateActions

apply throws an AnalysisException for other actions:

[name] clauses cannot be part of the WHEN NOT MATCHED BY SOURCE clause in MERGE INTO.

In the end, apply creates a DeltaMergeInto logical command (with the matched and not-matched actions).

OverwriteByExpression¶

OverwritePartitionsDynamic¶

RestoreTableStatement¶

UnresolvedPathBasedDeltaTable¶

UpdateTable¶

WriteToStream¶