DeltaMergeInto Logical Command¶

DeltaMergeInto is a logical Command (Spark SQL).

Creating Instance¶

DeltaMergeInto takes the following to be created:

• Target Table (Spark SQL)
• Source Table or Subquery (Spark SQL)
• Condition Expression
• DeltaMergeIntoMatchedClauses
• Non-Matched DeltaMergeIntoInsertClauses
• migrateSchema flag

When created, DeltaMergeInto verifies the actions in the matchedClauses and notMatchedClauses clauses.

DeltaMergeInto is created (using apply and resolveReferences utilities) when:

• DeltaMergeBuilder is requested to execute
• DeltaAnalysis logical resolution rule is executed

Logical Resolution¶

DeltaMergeInto is resolved to MergeIntoCommand by PreprocessTableMerge logical resolution rule.

migrateSchema Flag¶

DeltaMergeInto is given migrateSchema flag when created:

• apply uses false always
• resolveReferences is true only with the spark.databricks.delta.schema.autoMerge.enabled configuration property enabled and *s only (in matched and not-matched clauses)

migrateSchema is used when:

• PreprocessTableMerge logical resolution rule is executed

SupportsSubquery¶

DeltaMergeInto is a SupportsSubquery (Spark SQL)

Creating DeltaMergeInto¶

apply(
  target: LogicalPlan,
  source: LogicalPlan,
  condition: Expression,
  whenClauses: Seq[DeltaMergeIntoClause]): DeltaMergeInto

apply collects DeltaMergeIntoInsertClauses and DeltaMergeIntoMatchedClauses from the given whenClauses and creates a DeltaMergeInto command (with migrateSchema flag off).

apply is used when:

• DeltaMergeBuilder is requested to execute (when mergePlan)
• DeltaAnalysis logical resolution rule is executed (and resolves MergeIntoTable logical command)

AnalysisExceptions¶

apply throws an AnalysisException for the whenClauses empty:

There must be at least one WHEN clause in a MERGE statement

apply throws an AnalysisException if there is a matched clause with no condition (except the last matched clause):

When there are more than one MATCHED clauses in a MERGE statement,
only the last MATCHED clause can omit the condition.

apply throws an AnalysisException if there is an insert clause with no condition (except the last matched clause):

When there are more than one NOT MATCHED clauses in a MERGE statement,
only the last NOT MATCHED clause can omit the condition.

resolveReferencesAndSchema¶

resolveReferencesAndSchema(
  merge: DeltaMergeInto,
  conf: SQLConf)(
  resolveExpr: (Expression, LogicalPlan) => Expression): DeltaMergeInto

resolveReferencesAndSchema...FIXME

resolveReferencesAndSchema is used when:

• DeltaMergeBuilder is requested to execute
• DeltaAnalysis logical resolution rule is executed (to resolve MergeIntoTable logical command)