DeltaParquetFileFormat¶

DeltaParquetFileFormat is a ParquetFileFormat (Spark SQL) to support no restrictions on columns names.

Creating Instance¶

DeltaParquetFileFormat takes the following to be created:

• Protocol
• Metadata
• isSplittable Flag
• disablePushDowns Flag
• Optional Table Path (default: None (unspecified))
• Optional Broadcast variable with DeletionVectorDescriptorWithFilterTypes per URI (default: None (unspecified))
• Optional Broadcast variable with Hadoop Configuration (default: None (unspecified))

DeltaParquetFileFormat is created when:

• DeltaFileFormat is requested for the fileFormat
• CDCReaderImpl is requested for the scanIndex

isSplittable Flag¶

DeltaParquetFileFormat can be given isSplittable flag when created.

Unless given, isSplittable flag is enabled by default (to match the base ParquetFileFormat (Spark SQL)).

isSplittable is disabled (false) when:

• DeltaParquetFileFormat is requested to copyWithDVInfo and created with deletion vectors enabled
• DMLWithDeletionVectorsHelper is requested to replace a FileIndex

isSplittable is also used to buildReaderWithPartitionValues (to assert the configuration of this delta table).

disablePushDowns¶

DeltaParquetFileFormat can be given disablePushDowns flag when created.

disablePushDowns flag indicates whether this delta table supports predicate pushdown optimization or not for buildReaderWithPartitionValues to pass the filters down to the parquet data reader or not.

Unless given, disablePushDowns flag is disabled (false) by default.

disablePushDowns is enabled (true) when:

• DeltaParquetFileFormat is requested to copyWithDVInfo and created with deletion vectors enabled
• DMLWithDeletionVectorsHelper is requested to replace a FileIndex

__delta_internal_row_index Internal Metadata Column¶

DeltaParquetFileFormat defines __delta_internal_row_index name for the metadata column name with the index of a row within a file.

__delta_internal_row_index is an internal column.

When defined in the schema (of a delta table), DeltaParquetFileFormat creates an iteratorWithAdditionalMetadataColumns.

__delta_internal_row_index is used when:

• DMLWithDeletionVectorsHelper is requested to replace a FileIndex (in all the delta tables in a logical plan)
• DeletionVectorBitmapGenerator is requested to buildRowIndexSetsForFilesMatchingCondition

Building Data Reader (With Partition Values)¶

buildReaderWithPartitionValues(
  sparkSession: SparkSession,
  dataSchema: StructType,
  partitionSchema: StructType,
  requiredSchema: StructType,
  filters: Seq[Filter],
  options: Map[String, String],
  hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow]

buildReaderWithPartitionValues prepares the given schemas (e.g., dataSchema, partitionSchema and requiredSchema) before requesting the parent ParquetFileFormat to buildReaderWithPartitionValues.

buildReaderWithPartitionValues is part of the ParquetFileFormat (Spark SQL) abstraction.

Preparing Schema¶

prepareSchema(
  inputSchema: StructType): StructType

prepareSchema creates a physical schema (for the inputSchema, the referenceSchema and the DeltaColumnMappingMode).

supportFieldName¶

supportFieldName(
  name: String): Boolean

supportFieldName is enabled (true) when either holds true:

• The DeltaColumnMappingMode is not NoMapping
• The default (parent) supportFieldName (Spark SQL)

metadataSchemaFields¶

metadataSchemaFields: Seq[StructField]

Due to an issue in Spark SQL (to be reported), metadataSchemaFields removes row_index from the default metadataSchemaFields (Spark SQL).