ImplicitMetadataOperation¶
ImplicitMetadataOperation
is an abstraction of operations that can update metadata of a delta table (while writing out a new data).
ImplicitMetadataOperation
operations can update schema by merging and overwriting schema.
Contract¶
canMergeSchema¶
canMergeSchema: Boolean
Used when:
MergeIntoCommand
command is executedImplicitMetadataOperation
is requested to updateMetadata
canOverwriteSchema¶
canOverwriteSchema: Boolean
Used when:
ImplicitMetadataOperation
is requested to updateMetadata
Implementations¶
Updating Metadata¶
updateMetadata( // (1)
txn: OptimisticTransaction,
data: Dataset[_],
partitionColumns: Seq[String],
configuration: Map[String, String],
isOverwriteMode: Boolean,
rearrangeOnly: Boolean = false): Unit
updateMetadata(
spark: SparkSession,
txn: OptimisticTransaction,
schema: StructType,
partitionColumns: Seq[String],
configuration: Map[String, String],
isOverwriteMode: Boolean,
rearrangeOnly: Boolean): Unit
- Uses the
SparkSession
and the schema of the givenDataset
and assumes therearrangeOnly
to be off
updateMetadata
...FIXME
updateMetadata
is used when:
- MergeIntoCommand and WriteIntoDelta commands are executed
DeltaSink
is requested to add a streaming micro-batch
Normalizing Partition Columns¶
normalizePartitionColumns(
spark: SparkSession,
partitionCols: Seq[String],
schema: StructType): Seq[String]
normalizePartitionColumns
...FIXME