DeltaParquetFileFormatBase¶

DeltaParquetFileFormatBase is a base abstraction of the ParquetFileFormat (Spark SQL) abstraction for delta file formats (with data files in parquet file format).

Deletion Vectors-Enabled Scans¶

DeltaParquetFileFormatBase makes a precondition/invariant check for delta table scans with Deletion Vectors enabled.

When the delta table is referenced by a path (not a name or identifier) and useMetadataRowIndexOpt is defined, the flag should match optimizationsEnabled.

If they differ, DeltaParquetFileFormatBase throws an IllegalArgumentException:

Wrong arguments for Delta table scan with deletion vectors

useMetadataRowIndexOpt Flag¶

DeltaParquetFileFormatBase can be given useMetadataRowIndexOpt flag when created for row index source for Deletion Vectors filtering.

useMetadataRowIndexOpt flag is undefined (None) by default.

useMetadataRowIndexOpt is configured using spark.databricks.delta.deletionVectors.useMetadataRowIndex configuration property (for DeltaParquetFileFormat).

useMetadataRowIndexOpt is used for buildReaderWithPartitionValues, if defined, or defaults to spark.databricks.delta.deletionVectors.useMetadataRowIndex configuration property.

For path-based delta tables with useMetadataRowIndexOpt defined, the value must be exactly this optimizationsEnabled (for Deletion Vectors-Enabled Scans).

optimizationsEnabled Flag¶

DeltaParquetFileFormatBase can be given optimizationsEnabled flag when created to enable scan optimizations (file splitting and predicate pushdown).

In other words, optimizationsEnabled flag is equivalent to using scan optimizations (file splitting and predicate pushdown).

optimizationsEnabled is enabled (true) by default.

optimizationsEnabled controls isSplitable predicate.

optimizationsEnabled must be enabled when buildReaderWithPartitionValues with this useMetadataRowIndex flag disabled.

When optimizationsEnabled is disabled, prepareFiltersForRead gives no filters.

optimizationsEnabled must be exactly this useMetadataRowIndexOpt (when defined) for path-based delta tables.

optimizationsEnabled is used when:

• DeltaParquetFileFormatBase is created for Deletion Vectors-Enabled Scans
• DMLWithDeletionVectorsHelper is requested to replaceFileIndex (where this flag is disabled)

Implementations¶

• DeltaParquetFileFormat
• DeltaParquetFileFormatV2

Creating Instance¶

DeltaParquetFileFormatBase takes the following to be created:

• ProtocolMetadataAdapter
• nullableRowTrackingConstantFields flag (default: false)
• nullableRowTrackingGeneratedFields flag (default: false)
• optimizationsEnabled flag
• Optional Table Path (default: undefined)
• isCDCRead flag (default: false)
• useMetadataRowIndexOpt flag

While being created, DeltaParquetFileFormatBase asserts the following:

1. Deletion Vectors-Enabled Scans
2. delta table is readable.
3. Either nullableRowTrackingConstantFields is disabled or nullableRowTrackingGeneratedFields is enabled.
4. With columnMappingMode as IdMapping, ...FIXME

isSplitable¶

isSplitable(
  sparkSession: SparkSession,
  options: Map[String, String],
  path: Path): Boolean

isSplitable returns this optimizationsEnabled.

Build Data Reader (with Partition Values)¶

buildReaderWithPartitionValues(
  sparkSession: SparkSession,
  dataSchema: StructType,
  partitionSchema: StructType,
  requiredSchema: StructType,
  filters: Seq[Filter],
  options: Map[String, String],
  hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow]

buildReaderWithPartitionValues...FIXME

prepareSchemaForRead¶

prepareSchemaForRead(
  inputSchema: StructType): StructType

prepareSchemaForRead...FIXME

prepareFiltersForRead¶

prepareFiltersForRead(
  filters: Seq[Filter]): Seq[Filter]

prepareFiltersForRead...FIXME

hasTablePath¶

hasTablePath: Boolean

hasTablePath is enabled (true) when this delta table is referenced by a path (not a name/identifier).

hasTablePath is used when:

• DeltaParquetFileFormatBase is created and requested to buildReaderWithPartitionValues

supportFieldName¶

supportFieldName(
  name: String): Boolean

supportFieldName is enabled (true) when either holds true:

• DeltaColumnMappingMode is not NoMapping
• The default (parent) supportFieldName (Spark SQL) is enabled

metadataSchemaFields¶

metadataSchemaFields: Seq[StructField]

Due to an issue in Spark SQL (to be reported), metadataSchemaFields removes row_index from the default metadataSchemaFields (Spark SQL).

prepareWrite¶

prepareWrite(
   sparkSession: SparkSession,
   job: Job,
   options: Map[String, String],
   dataSchema: StructType): OutputWriterFactory

prepareWrite...FIXME

fileConstantMetadataExtractors¶

fileConstantMetadataExtractors: Map[String, PartitionedFile => Any]

fileConstantMetadataExtractors...FIXME