DeltaDynamicPartitionOverwriteCommand¶

DeltaDynamicPartitionOverwriteCommand is a RunnableCommand (Spark SQL) and a V2WriteCommand (Spark SQL) for dynamic partition overwrite using WriteIntoDelta.

DeltaDynamicPartitionOverwriteCommand sets partitionOverwriteMode option as DYNAMIC before write.

Workaround

DeltaDynamicPartitionOverwriteCommand is a workaround of Spark SQL not supporting V1 fallback for dynamic partition overwrite.

Creating Instance¶

DeltaDynamicPartitionOverwriteCommand takes the following to be created:

Delta Table (NamedRelation)
DeltaTableV2
Logical Query Plan (LogicalPlan)
Write options (Map[String, String])
isByName flag
Analyzed Logical Query Plan (Option[LogicalPlan])

DeltaDynamicPartitionOverwriteCommand is created when:

DeltaAnalysis logical resolution rule is requested to resolve an OverwritePartitionsDynamic on a delta table (INSERT OVERWRITE with dynamic partition overwrite)

OverwritePartitionsDynamic Unary Logical Command

Learn more about OverwritePartitionsDynamic unary logical command in The Internals of Spark SQL.

Executing Command¶

RunnableCommand

run(
  sparkSession: SparkSession): Seq[Row]

run is part of the RunnableCommand (Spark SQL) abstraction.

run...FIXME