Skip to content

ImplicitMetadataOperation

ImplicitMetadataOperation is an abstraction of operations that can update metadata of a delta table (while writing out a new data).

ImplicitMetadataOperation operations can update schema by merging and overwriting schema.

Contract

canMergeSchema

canMergeSchema: Boolean

Used when:

canOverwriteSchema

canOverwriteSchema: Boolean

Used when:

Implementations

Updating Metadata

updateMetadata( // (1)
  txn: OptimisticTransaction,
  data: Dataset[_],
  partitionColumns: Seq[String],
  configuration: Map[String, String],
  isOverwriteMode: Boolean,
  rearrangeOnly: Boolean = false): Unit
updateMetadata(
  spark: SparkSession,
  txn: OptimisticTransaction,
  schema: StructType,
  partitionColumns: Seq[String],
  configuration: Map[String, String],
  isOverwriteMode: Boolean,
  rearrangeOnly: Boolean): Unit
  1. Uses the SparkSession and the schema of the given Dataset and assumes the rearrangeOnly to be off

updateMetadata...FIXME

updateMetadata is used when:

Normalizing Partition Columns

normalizePartitionColumns(
  spark: SparkSession,
  partitionCols: Seq[String],
  schema: StructType): Seq[String]

normalizePartitionColumns...FIXME


Last update: 2021-03-22