DeltaColumnMappingBase (DeltaColumnMapping)¶
DeltaColumnMappingBase
is an abstraction of DeltaColumnMappings.
Implementations¶
createPhysicalSchema¶
createPhysicalSchema(
schema: StructType,
referenceSchema: StructType,
columnMappingMode: DeltaColumnMappingMode,
checkSupportedMode: Boolean = true): StructType
createPhysicalSchema
...FIXME
createPhysicalSchema
is used when:
DeltaColumnMappingBase
is requested to checkColumnIdAndPhysicalNameAssignments and createPhysicalAttributesDeltaParquetFileFormat
is requested to prepare a schema
renameColumns¶
renameColumns(
schema: StructType): StructType
renameColumns
...FIXME
renameColumns
is used when:
Metadata
is requested for the physicalPartitionSchema
requiresNewProtocol¶
requiresNewProtocol(
metadata: Metadata): Boolean
requiresNewProtocol
...FIXME
requiresNewProtocol
is used when:
Protocol
utility is used for the required minimum protocol
checkColumnIdAndPhysicalNameAssignments¶
checkColumnIdAndPhysicalNameAssignments(
schema: StructType,
mode: DeltaColumnMappingMode): Unit
checkColumnIdAndPhysicalNameAssignments
...FIXME
checkColumnIdAndPhysicalNameAssignments
is used when:
OptimisticTransactionImpl
is requested to verify the new metadata
dropColumnMappingMetadata¶
dropColumnMappingMetadata(
schema: StructType): StructType
dropColumnMappingMetadata
...FIXME
dropColumnMappingMetadata
is used when:
DeltaLog
is requested for a BaseRelation and for a DataFrameDeltaTableV2
is requested for the tableSchema- AlterTableSetLocationDeltaCommand command is executed
- CreateDeltaTableCommand command is executed
ImplicitMetadataOperation
is requested to update the metadata
Mapping Virtual to Physical Field Name¶
getPhysicalName(
field: StructField): String
getPhysicalName
requests the given StructField
(Spark SQL) for the Metadata
to extract delta.columnMapping.physicalName
key, if available (for column mapping). Otherwise, getPhysicalName
returns the name of the given StructField
(with no name changes).
getPhysicalName
is used when:
CheckpointV2
utility is used to extractPartitionValuesConflictChecker
is requested to getPrettyPartitionMessageDeltaColumnMappingBase
is requested to renameColumns, assignPhysicalNames and createPhysicalSchemaDeltaLog
utility is used to rewritePartitionFilters- AlterTableChangeColumnDeltaCommand is executed
ConvertToDeltaCommand
utility is used to create an AddFileTahoeFileIndex
is requested to makePartitionDirectoriesDataSkippingReaderBase
is requested to getStatsColumnOptStatisticsCollection
is requested to collect statistics
verifyAndUpdateMetadataChange¶
verifyAndUpdateMetadataChange(
oldProtocol: Protocol,
oldMetadata: Metadata,
newMetadata: Metadata,
isCreatingNewTable: Boolean): Metadata
verifyAndUpdateMetadataChange
...FIXME
In the end, verifyAndUpdateMetadataChange
tryFixMetadata with the given newMetadata
and oldMetadata
metadata.
verifyAndUpdateMetadataChange
is used when:
OptimisticTransactionImpl
is requested to updateMetadataInternal
tryFixMetadata¶
tryFixMetadata(
oldMetadata: Metadata,
newMetadata: Metadata,
isChangingModeOnExistingTable: Boolean): Metadata
tryFixMetadata
reads columnMapping.mode table property from the given newMetadata
table metadata.
If the DeltaColumnMappingMode is IdMapping or NameMapping, tryFixMetadata
assignColumnIdAndPhysicalName with the given newMetadata
and oldMetadata
metadata and isChangingModeOnExistingTable
flag.
For NoMapping
, tryFixMetadata
does nothing and returns the given newMetadata
.
DeltaColumnMapping¶
DeltaColumnMapping
is the only DeltaColumnMappingBase.