DeltaMergeInto Logical Command¶
DeltaMergeInto
is a logical Command
(Spark SQL).
Creating Instance¶
DeltaMergeInto
takes the following to be created:
- Target Table (Spark SQL)
- Source Table or Subquery (Spark SQL)
- Condition Expression
- DeltaMergeIntoMatchedClauses
- Non-Matched DeltaMergeIntoInsertClauses
- migrateSchema flag
When created, DeltaMergeInto
verifies the actions in the matchedClauses and notMatchedClauses clauses.
DeltaMergeInto
is created (using apply and resolveReferences utilities) when:
DeltaMergeBuilder
is requested to execute- DeltaAnalysis logical resolution rule is executed
Logical Resolution¶
DeltaMergeInto
is resolved to MergeIntoCommand by PreprocessTableMerge logical resolution rule.
migrateSchema Flag¶
DeltaMergeInto
is given migrateSchema
flag when created:
- apply uses
false
always - resolveReferences is
true
only with the spark.databricks.delta.schema.autoMerge.enabled configuration property enabled and*
s only (in matched and not-matched clauses)
migrateSchema
is used when:
- PreprocessTableMerge logical resolution rule is executed
SupportsSubquery¶
DeltaMergeInto
is a SupportsSubquery
(Spark SQL)
Creating DeltaMergeInto¶
apply(
target: LogicalPlan,
source: LogicalPlan,
condition: Expression,
whenClauses: Seq[DeltaMergeIntoClause]): DeltaMergeInto
apply
collects DeltaMergeIntoInsertClauses and DeltaMergeIntoMatchedClauses from the given whenClauses
and creates a DeltaMergeInto
command (with migrateSchema flag off).
apply
is used when:
DeltaMergeBuilder
is requested to execute (when mergePlan)- DeltaAnalysis logical resolution rule is executed (and resolves
MergeIntoTable
logical command)
AnalysisExceptions¶
apply
throws an AnalysisException
for the whenClauses
empty:
There must be at least one WHEN clause in a MERGE statement
apply
throws an AnalysisException
if there is a matched clause with no condition (except the last matched clause):
When there are more than one MATCHED clauses in a MERGE statement,
only the last MATCHED clause can omit the condition.
apply
throws an AnalysisException
if there is an insert clause with no condition (except the last matched clause):
When there are more than one NOT MATCHED clauses in a MERGE statement,
only the last NOT MATCHED clause can omit the condition.
resolveReferencesAndSchema¶
resolveReferencesAndSchema(
merge: DeltaMergeInto,
conf: SQLConf)(
resolveExpr: (Expression, LogicalPlan) => Expression): DeltaMergeInto
resolveReferencesAndSchema
...FIXME
resolveReferencesAndSchema
is used when:
DeltaMergeBuilder
is requested to execute- DeltaAnalysis logical resolution rule is executed (to resolve
MergeIntoTable
logical command)