Dynamic Partition Overwrite¶
Dynamic Partition Overwrite is a DataFrameWriter
(Spark SQL) feature that allows to only overwrite partitions (of a partitioned delta table) that have data written into it.
Dynamic Partition Overwrite can be enabled system-wide using spark.databricks.delta.dynamicPartitionOverwrite.enabled configuration property.
DeltaIllegalArgumentException
It is a DeltaIllegalArgumentException for spark.databricks.delta.dynamicPartitionOverwrite.enabled configuration property disabled yet the Dynamic Partition Overwrite Mode is dynamic.
Conflicts with replaceWhere
Dynamic Partition Overwrite cannot be used with replaceWhere option as they both specify which data to overwrite.
Partition Overwrite Mode¶
Partition Overwrite Mode can be one of the following values (case-insensitive):