Skip to content

DeltaMergeInto Logical Command

DeltaMergeInto is a logical Command (Spark SQL).

Creating Instance

DeltaMergeInto takes the following to be created:

When created, DeltaMergeInto verifies the actions in the matchedClauses and notMatchedClauses clauses.

DeltaMergeInto is created (using apply and resolveReferences utilities) when:

PreprocessTableMerge

DeltaMergeInto is resolved to MergeIntoCommand by PreprocessTableMerge logical resolution rule.

migrateSchema Flag

DeltaMergeInto is given migrateSchema flag when created:

migrateSchema is used when:

SupportsSubquery

DeltaMergeInto is a SupportsSubquery (Spark SQL)

Creating DeltaMergeInto

apply(
  target: LogicalPlan,
  source: LogicalPlan,
  condition: Expression,
  whenClauses: Seq[DeltaMergeIntoClause]): DeltaMergeInto

apply collects DeltaMergeIntoInsertClauses and DeltaMergeIntoMatchedClauses from the given whenClauses and creates a DeltaMergeInto command (with migrateSchema flag off).

apply throws an AnalysisException for the whenClauses empty:

There must be at least one WHEN clause in a MERGE statement

apply throws an AnalysisException if there is a matched clause with no condition (except the last matched clause):

When there are more than one MATCHED clauses in a MERGE statement, only the last MATCHED clause can omit the condition.

apply throws an AnalysisException if there is an insert clause with no condition (except the last matched clause):

When there are more than one NOT MATCHED clauses in a MERGE statement, only the last NOT MATCHED clause can omit the condition.

apply is used when:

  • DeltaMergeBuilder is requested to execute (when mergePlan)
  • DeltaAnalysis logical resolution rule is executed (and resolves MergeIntoTable logical command)

resolveReferences

resolveReferences(
  merge: DeltaMergeInto,
  conf: SQLConf)(
  resolveExpr: (Expression, LogicalPlan) => Expression): DeltaMergeInto

resolveReferences...FIXME

resolveReferences is used when:

  • DeltaMergeBuilder is requested to execute
  • DeltaAnalysis logical resolution rule is executed (and resolves MergeIntoTable logical command)

Last update: 2021-06-01
Back to top