Skip to content


TahoeFileIndex is an extension of the FileIndex abstraction (Spark SQL) for file indices of delta tables that can list data files to scan (based on partition and data filters).

The aim of TahoeFileIndex (and FileIndex in general) is to reduce usage of very expensive disk access for file-related information using Hadoop FileSystem API.



  partitionFilters: Seq[Expression],
  dataFilters: Seq[Expression]): Seq[AddFile]

AddFiles matching given partition and data filters (predicates)

Used for listing data files


Creating Instance

TahoeFileIndex takes the following to be created:

Abstract Class

TahoeFileIndex is an abstract class and cannot be created directly. It is created indirectly for the concrete TahoeFileIndexes.

Root Paths

rootPaths: Seq[Path]

rootPaths is the path only.

rootPaths is part of the FileIndex abstraction (Spark SQL).

Listing Files

  partitionFilters: Seq[Expression],
  dataFilters: Seq[Expression]): Seq[PartitionDirectory]

listFiles is the path only.

listFiles is part of the FileIndex abstraction (Spark SQL).


partitionSchema: StructType

partitionSchema is the partition schema of (the Metadata of the Snapshot) of the DeltaLog.

partitionSchema is part of the FileIndex abstraction (Spark SQL).

Version of Delta Table

tableVersion: Long

tableVersion is the version of (the snapshot of) the DeltaLog.

tableVersion is used when TahoeFileIndex is requested for the human-friendly textual representation.

Textual Representation

toString: String

toString returns the following text (using the version and the path of the Delta table):

Delta[version=[tableVersion], [truncatedPath]]

toString is part of the java.lang.Object contract for a string representation of the object.

Last update: 2020-11-19