DeltaColumnMappingBase (DeltaColumnMapping)¶

DeltaColumnMappingBase is an abstraction of DeltaColumnMappings.

Implementations¶

DeltaColumnMapping

createPhysicalSchema¶

createPhysicalSchema(
  schema: StructType,
  referenceSchema: StructType,
  columnMappingMode: DeltaColumnMappingMode,
  checkSupportedMode: Boolean = true): StructType

createPhysicalSchema...FIXME

createPhysicalSchema is used when:

DeltaColumnMappingBase is requested to checkColumnIdAndPhysicalNameAssignments and createPhysicalAttributes
DeltaParquetFileFormat is requested to prepare a schema

renameColumns¶

renameColumns(
  schema: StructType): StructType

renameColumns...FIXME

renameColumns is used when:

Metadata is requested for the physicalPartitionSchema

requiresNewProtocol¶

requiresNewProtocol(
  metadata: Metadata): Boolean

requiresNewProtocol...FIXME

requiresNewProtocol is used when:

Protocol utility is used for the required minimum protocol

checkColumnIdAndPhysicalNameAssignments¶

checkColumnIdAndPhysicalNameAssignments(
  schema: StructType,
  mode: DeltaColumnMappingMode): Unit

checkColumnIdAndPhysicalNameAssignments...FIXME

checkColumnIdAndPhysicalNameAssignments is used when:

OptimisticTransactionImpl is requested to verify the new metadata

dropColumnMappingMetadata¶

dropColumnMappingMetadata(
  schema: StructType): StructType

dropColumnMappingMetadata...FIXME

dropColumnMappingMetadata is used when:

DeltaLog is requested for a BaseRelation and for a DataFrame
DeltaTableV2 is requested for the tableSchema
AlterTableSetLocationDeltaCommand command is executed
CreateDeltaTableCommand command is executed
ImplicitMetadataOperation is requested to update the metadata

Mapping Virtual to Physical Field Name¶

getPhysicalName(
  field: StructField): String

getPhysicalName requests the given StructField (Spark SQL) for the Metadata to extract delta.columnMapping.physicalName key, if available (for column mapping). Otherwise, getPhysicalName returns the name of the given StructField (with no name changes).

getPhysicalName is used when:

CheckpointV2 utility is used to extractPartitionValues
ConflictChecker is requested to getPrettyPartitionMessage
DeltaColumnMappingBase is requested to renameColumns, assignPhysicalNames and createPhysicalSchema
DeltaLog utility is used to rewritePartitionFilters
AlterTableChangeColumnDeltaCommand is executed
ConvertToDeltaCommand utility is used to create an AddFile
TahoeFileIndex is requested to makePartitionDirectories
DataSkippingReaderBase is requested to getStatsColumnOpt
StatisticsCollection is requested to collect statistics

verifyAndUpdateMetadataChange¶

verifyAndUpdateMetadataChange(
  oldProtocol: Protocol,
  oldMetadata: Metadata,
  newMetadata: Metadata,
  isCreatingNewTable: Boolean): Metadata

verifyAndUpdateMetadataChange...FIXME

In the end, verifyAndUpdateMetadataChange tryFixMetadata with the given newMetadata and oldMetadata metadata.

verifyAndUpdateMetadataChange is used when:

OptimisticTransactionImpl is requested to updateMetadataInternal

tryFixMetadata¶

tryFixMetadata(
  oldMetadata: Metadata,
  newMetadata: Metadata,
  isChangingModeOnExistingTable: Boolean): Metadata

tryFixMetadata reads columnMapping.mode table property from the given newMetadata table metadata.

If the DeltaColumnMappingMode is IdMapping or NameMapping, tryFixMetadata assignColumnIdAndPhysicalName with the given newMetadata and oldMetadata metadata and isChangingModeOnExistingTable flag.

For NoMapping, tryFixMetadata does nothing and returns the given newMetadata.

DeltaColumnMapping¶

DeltaColumnMapping is the only DeltaColumnMappingBase.