DeltaWriteOptionsImpl¶

DeltaWriteOptionsImpl is an extension of the DeltaOptionParser abstraction.

Fun Fact

Despite the suffix (Impl), DeltaWriteOptionsImpl is a trait not an implementation (class).

Auto Schema Merging¶

canMergeSchema: Boolean

canMergeSchema flag controls Auto Schema Merging based on mergeSchema option, if defined, or schema.autoMerge.enabled configuration property.

canMergeSchema is used when:

MergeIntoCommandBase is requested for canMergeSchema (while running a merge)
WriteIntoDelta is created
DeltaSink is created

canOverwriteSchema¶

canOverwriteSchema: Boolean

canOverwriteSchema is the value of overwriteSchema option (in the options).

canOverwriteSchema is used when:

CreateDeltaTableCommand is executed (and replaceMetadataIfNecessary)
WriteIntoDelta is created
DeltaSink is created

rearrangeOnly¶

rearrangeOnly: Boolean

rearrangeOnly is the negation of the value of dataChange option.

rearrangeOnly is used when:

WriteIntoDelta is requested to write

isDynamicPartitionOverwriteMode¶

isDynamicPartitionOverwriteMode: Boolean

isDynamicPartitionOverwriteMode determines the partition overwrite mode based on the value of partitionOverwriteMode option (in the options), if specified, or defaults to the value of spark.sql.sources.partitionOverwriteMode (Spark SQL).

DeltaIllegalArgumentException: DYNAMIC mode with Dynamic Partition Overwrite disabled

For DYNAMIC partition overwrite mode and dynamicPartitionOverwrite.enabled disabled, isDynamicPartitionOverwriteMode reports a DeltaIllegalArgumentException.

With dynamicPartitionOverwrite.enabled disabled and the partition overwrite mode is anything but DYNAMIC, isDynamicPartitionOverwriteMode is off (returns false).

Otherwise, isDynamicPartitionOverwriteMode is whether the partition overwrite mode is dynamic or not (static).

isDynamicPartitionOverwriteMode is used when:

WriteIntoDelta is requested to write data out