DeltaLogFileIndex

DeltaLogFileIndex is a FileIndex (Spark SQL) that takes the following to be created:

  • One of the supported FileFormats

  • Array[FileStatus] (Hadoop)

DeltaLogFileIndex is created (directly or indirectly using apply utility) when DeltaLog is requested for the currentSnapshot, to updateInternal and getSnapshotAt.

apply Utility

apply(
  format: FileFormat,
  fs: FileSystem,
  paths: Seq[Path]): DeltaLogFileIndex

apply simply creates a new DeltaLogFileIndex.

apply is used when DeltaLog is requested for the currentSnapshot, to updateInternal and getSnapshotAt.