LogStore¶

LogStore is an abstraction of transaction log stores (to read and write physical log files and checkpoints).

LogStore is created using LogStoreProvider based on spark.delta.logStore.class configuration property.

Contract¶

invalidateCache(): Unit

Used when:

isPartialWriteVisible(
  path: Path): Boolean // (1)!
isPartialWriteVisible(
  path: Path,
  hadoopConf: Configuration): Boolean

Whether a partial write is visible when writing to path

Default: true

Used when:

listFrom(
  path: Path): Iterator[FileStatus]
listFrom(
  path: String): Iterator[FileStatus]

Used when:

Checkpoints is requested to findLastCompleteCheckpoint
DeltaHistoryManager is requested to getEarliestDeltaFile, getEarliestReproducibleCommit and getCommits
DeltaLog is requested to getChanges
MetadataCleanup is requested to listExpiredDeltaLogs
SnapshotManagement is requested to listFrom
DelegatingLogStore is requested to listFrom
DeltaFileOperations utility is used to listUsingLogStore

read(
  path: Path): Seq[String]
read(
  path: String): Seq[String]

Used when:

write(
  path: Path,
  actions: Iterator[String],
  overwrite: Boolean = false): Unit
write(
  path: String,
  actions: Iterator[String]): Unit

Used when:

apply(
  sc: SparkContext): LogStore
apply(
  sparkConf: SparkConf,
  hadoopConf: Configuration): LogStore

apply is used when:

GenerateSymlinkManifestImpl is requested to writeManifestFiles and writeSingleManifestFile
DeltaHistoryManager is requested to getHistory and getActiveCommitAtTime
DeltaFileOperations is requested to recursiveListDirs, localListDirs, and localListFrom

createLogStoreWithClassName(
  className: String,
  sparkConf: SparkConf,
  hadoopConf: Configuration): LogStore

createLogStoreWithClassName branches off based on the given className.

createLogStoreWithClassName creates a DelegatingLogStore when the className is the fully-qualified class name of DelegatingLogStore.

Otherwise, createLogStoreWithClassName loads the class and braches off based on the class type.

For io.delta.storage.LogStore, createLogStoreWithClassName creates an instance thereof (with the given Configuration) and wraps it up in a LogStoreAdaptor.

For all other cases, createLogStoreWithClassName creates an instance thereof (with the given SparkConf and Configuration).

createLogStoreWithClassName is used when:

logStoreSchemeConfKey(
  scheme: String): String

logStoreSchemeConfKey simply returns the following text for the given scheme:

spark.delta.logStore.[scheme].impl

logStoreSchemeConfKey is used when: