DeltaFileOperations Utilities¶

listUsingLogStore¶

listUsingLogStore(
  logStore: LogStore,
  subDirs: Iterator[String],
  recurse: Boolean,
  hiddenFileNameFilter: String => Boolean): Iterator[SerializableFileStatus]

listUsingLogStore...FIXME

listUsingLogStore is used when:

DeltaFileOperations utility is used to recurseDirectories, recursiveListDirs and localListDirs

localListDirs¶

localListDirs(
  spark: SparkSession,
  dirs: Seq[String],
  recursive: Boolean = true,
  fileFilter: String => Boolean = defaultHiddenFileFilter): Seq[SerializableFileStatus]

localListDirs...FIXME

localListDirs seems not used.

recurseDirectories¶

recurseDirectories(
  logStore: LogStore,
  filesAndDirs: Iterator[SerializableFileStatus],
  hiddenFileNameFilter: String => Boolean): Iterator[SerializableFileStatus]

recurseDirectories...FIXME

recurseDirectories is used when:

DeltaFileOperations utility is used to listUsingLogStore and recursiveListDirs

recursiveListDirs¶

recursiveListDirs(
  spark: SparkSession,
  subDirs: Seq[String],
  hadoopConf: Broadcast[SerializableConfiguration],
  hiddenFileNameFilter: String => Boolean = defaultHiddenFileFilter,
  fileListingParallelism: Option[Int] = None): Dataset[SerializableFileStatus]

recursiveListDirs...FIXME

recursiveListDirs is used when:

ManualListingFileManifest is requested to doList
VacuumCommand utility is used to gc

tryDeleteNonRecursive¶

tryDeleteNonRecursive(
  fs: FileSystem,
  path: Path,
  tries: Int = 3): Boolean

tryDeleteNonRecursive...FIXME

tryDeleteNonRecursive is used when:

VacuumCommandImpl is requested to delete

Last update: 2021-04-24