DeltaParquetFileFormat¶

DeltaParquetFileFormat is a ParquetFileFormat (Spark SQL) to support no restrictions on columns names.

Creating Instance¶

DeltaParquetFileFormat takes the following to be created:

• Protocol
• Metadata
• nullableRowTrackingFields flag (default: false)
• optimizationsEnabled flag (default: true)
• Optional Table Path (default: None (unspecified))
• isCDCRead flag (default: false)

DeltaParquetFileFormat is created when:

• DeltaFileFormat is requested for the file format
• CDCReaderImpl is requested for the scanIndex

__delta_internal_row_index Internal Metadata Column¶

DeltaParquetFileFormat defines __delta_internal_row_index name for the metadata column name with the index of a row within a file.

__delta_internal_row_index is an internal column.

__delta_internal_row_index is used when:

• DMLWithDeletionVectorsHelper is requested to replace a FileIndex (in all the delta tables in a logical plan)
• DeletionVectorBitmapGenerator is requested to buildRowIndexSetsForFilesMatchingCondition