MergeIntoCommand

MergeIntoCommand is a DeltaCommand that can…​FIXME

MergeIntoCommand is a logical command (Spark SQL’s RunnableCommand).

Read up on RunnableCommand in The Internals of Spark SQL online book.

Creating Instance

MergeIntoCommand takes the following to be created:

  • Source Data (LogicalPlan)

  • Target Data (LogicalPlan)

  • TahoeFileIndex — Indices Of Files Of Delta Table

  • Condition Expression

  • Matched Clauses (Seq[DeltaMergeIntoMatchedClause])

  • Optional Non-Matched Clause (Option[DeltaMergeIntoInsertClause])

  • Optional Migrated Schema

MergeIntoCommand is created when PreprocessTableMerge logical resolution rule is requested to execute.

Running Command

run(
  spark: SparkSession): Seq[Row]

run…​FIXME

run is part of the RunnableCommand (Spark SQL) abstraction.

writeAllChanges Internal Method

writeAllChanges(
  spark: SparkSession,
  deltaTxn: OptimisticTransaction,
  filesToRewrite: Seq[AddFile]): Seq[AddFile]

writeAllChanges…​FIXME

writeAllChanges is used when MergeIntoCommand is requested to run.

findTouchedFiles Internal Method

findTouchedFiles(
  deltaTxn: OptimisticTransaction,
  files: Seq[AddFile]): LogicalPlan

findTouchedFiles…​FIXME

findTouchedFiles is used when MergeIntoCommand is requested to run.

buildTargetPlanWithFiles Internal Method

buildTargetPlanWithFiles(
  deltaTxn: OptimisticTransaction,
  files: Seq[AddFile]): LogicalPlan

buildTargetPlanWithFiles…​FIXME

buildTargetPlanWithFiles is used when MergeIntoCommand is requested to run (via findTouchedFiles and writeAllChanges).