Skip to content

CheckpointV2

Extracting Partition Values

extractPartitionValues(
  partitionSchema: StructType,
  partitionValuesColName: String): Option[Column]
Noop for no partitionSchema

For no partitionSchema, extractPartitionValues does nothing and returns None (an undefined value).

For every field in the given partitionSchema, extractPartitionValues getPhysicalName and creates a Column with an expression.

FIXME Elaborate on 'an expression' 🙏

In the end, extractPartitionValues creates a struct of the extracted partition values as partitionValues_parsed column name.

extractPartitionValues is used when:

partitionValues_parsed Column Name

CheckpointV2 defines partitionValues_parsed as the name of the struct column when extracting partition values.