Skip to content

Dynamic Partition Overwrite

Dynamic Partition Overwrite is a DataFrameWriter (Spark SQL) feature that allows to only overwrite partitions (of a partitioned delta table) that have data written into it.

Dynamic Partition Overwrite can be enabled system-wide using spark.databricks.delta.dynamicPartitionOverwrite.enabled configuration property.

DeltaIllegalArgumentException

It is a DeltaIllegalArgumentException for spark.databricks.delta.dynamicPartitionOverwrite.enabled configuration property disabled yet the Dynamic Partition Overwrite Mode is dynamic.

Conflicts with replaceWhere

Dynamic Partition Overwrite cannot be used with replaceWhere option as they both specify which data to overwrite.

Partition Overwrite Mode

Partition Overwrite Mode can be one of the following values (case-insensitive):