ImplicitMetadataOperation¶

ImplicitMetadataOperation is an abstraction of operations that can update metadata of a delta table (while writing out a new data).

ImplicitMetadataOperation operations can update schema by merging and overwriting schema.

Contract¶

canMergeSchema¶

canMergeSchema: Boolean

Used when:

• MergeIntoCommand command is executed
• ImplicitMetadataOperation is requested to updateMetadata

canOverwriteSchema¶

canOverwriteSchema: Boolean

Used when:

• ImplicitMetadataOperation is requested to updateMetadata

Implementations¶

• DeltaSink
• MergeIntoCommand
• WriteIntoDelta

Updating Metadata¶

updateMetadata( // (1)
  txn: OptimisticTransaction,
  data: Dataset[_],
  partitionColumns: Seq[String],
  configuration: Map[String, String],
  isOverwriteMode: Boolean,
  rearrangeOnly: Boolean = false): Unit
updateMetadata(
  spark: SparkSession,
  txn: OptimisticTransaction,
  schema: StructType,
  partitionColumns: Seq[String],
  configuration: Map[String, String],
  isOverwriteMode: Boolean,
  rearrangeOnly: Boolean): Unit

1. Uses the SparkSession and the schema of the given Dataset and assumes the rearrangeOnly to be off

updateMetadata...FIXME

updateMetadata is used when:

• MergeIntoCommand and WriteIntoDelta commands are executed
• DeltaSink is requested to add a streaming micro-batch

Normalizing Partition Columns¶

normalizePartitionColumns(
  spark: SparkSession,
  partitionCols: Seq[String],
  schema: StructType): Seq[String]

normalizePartitionColumns...FIXME