Skip to content

DeltaFileOperations Utilities

listUsingLogStore

listUsingLogStore(
  logStore: LogStore,
  subDirs: Iterator[String],
  recurse: Boolean,
  hiddenFileNameFilter: String => Boolean): Iterator[SerializableFileStatus]

listUsingLogStore...FIXME

listUsingLogStore is used when:

localListDirs

localListDirs(
  spark: SparkSession,
  dirs: Seq[String],
  recursive: Boolean = true,
  fileFilter: String => Boolean = defaultHiddenFileFilter): Seq[SerializableFileStatus]

localListDirs...FIXME

localListDirs seems not used.

recurseDirectories

recurseDirectories(
  logStore: LogStore,
  filesAndDirs: Iterator[SerializableFileStatus],
  hiddenFileNameFilter: String => Boolean): Iterator[SerializableFileStatus]

recurseDirectories...FIXME

recurseDirectories is used when:

recursiveListDirs

recursiveListDirs(
  spark: SparkSession,
  subDirs: Seq[String],
  hadoopConf: Broadcast[SerializableConfiguration],
  hiddenFileNameFilter: String => Boolean = defaultHiddenFileFilter,
  fileListingParallelism: Option[Int] = None): Dataset[SerializableFileStatus]

recursiveListDirs...FIXME

recursiveListDirs is used when:

  • ManualListingFileManifest is requested to doList
  • VacuumCommand utility is used to gc

tryDeleteNonRecursive

tryDeleteNonRecursive(
  fs: FileSystem,
  path: Path,
  tries: Int = 3): Boolean

tryDeleteNonRecursive...FIXME

tryDeleteNonRecursive is used when:

  • VacuumCommandImpl is requested to delete

Last update: 2021-04-24
Back to top