ImplicitMetadataOperation¶

ImplicitMetadataOperation is an abstraction of operations that can update the metadata of a delta table (while writing out a new data).

ImplicitMetadataOperation operations can update schema by merging and overwriting schema.

Contract¶

canMergeSchema¶

canMergeSchema: Boolean

Used when:

• ImplicitMetadataOperation is requested to update the metadata

canOverwriteSchema¶

canOverwriteSchema: Boolean

Used when:

• ImplicitMetadataOperation is requested to update the metadata

Implementations¶

• DeltaSink
• MergeIntoCommand
• WriteIntoDelta

Updating Metadata¶

updateMetadata(
  spark: SparkSession,
  txn: OptimisticTransaction,
  schema: StructType,
  partitionColumns: Seq[String],
  configuration: Map[String, String],
  isOverwriteMode: Boolean,
  rearrangeOnly: Boolean): Unit

updateMetadata dropColumnMappingMetadata from the given schema (that produces dataSchema).

updateMetadata mergeSchema (with the dataSchema and the isOverwriteMode and canOverwriteSchema flags).

updateMetadata normalizePartitionColumns.

updateMetadata branches off based on the following conditions:

1. Delta table is just being created
2. Overwriting schema is enabled (i.e. isOverwriteMode and canOverwriteSchema flags are enabled, and either the schema is new or partitioning changed)
3. Merging schema is enabled the schema is new and the canMergeSchema is enabled (but the partitioning has not changed)
4. Data or Partitioning Schema has changed

Table Being Created¶

updateMetadata creates a new Metadata with the following:

• Uses the value of comment key (in the configuration) for the description
• FIXME

updateMetadata requests the given OptimisticTransaction to updateMetadata.

Overwriting Schema¶

updateMetadata...FIXME

Merging Schema¶

updateMetadata...FIXME

New Data or Partitioning Schema¶

updateMetadata...FIXME

isOverwriteMode¶

updateMetadata is given isOverwriteMode flag as follows:

• Only false for MergeIntoCommand with canMergeSchema enabled
• true for WriteIntoDelta in Overwrite save mode; false otherwise
• true for DeltaSink in Complete output mode; false otherwise

rearrangeOnly¶

updateMetadata is given rearrangeOnly flag as follows:

• Only false for MergeIntoCommand with canMergeSchema enabled
• rearrangeOnly option for WriteIntoDelta
• false for DeltaSink

configuration¶

updateMetadata is given configuration as follows:

• The existing configuration (of the metadata of the transaction) for MergeIntoCommand with canMergeSchema enabled
• configuration of the WriteIntoDelta command (while writing out)
• Always empty for DeltaSink

Usage¶

updateMetadata is used when:

• MergeIntoCommand command is executed (with canMergeSchema is enabled)
• WriteIntoDelta command is requested to write
• DeltaSink is requested to add a streaming micro-batch

Normalizing Partition Columns¶

normalizePartitionColumns(
  spark: SparkSession,
  partitionCols: Seq[String],
  schema: StructType): Seq[String]

normalizePartitionColumns...FIXME

mergeSchema¶

mergeSchema(
  txn: OptimisticTransaction,
  dataSchema: StructType,
  isOverwriteMode: Boolean,
  canOverwriteSchema: Boolean): StructType

mergeSchema...FIXME