Skip to content

LogStore

LogStore is an abstraction of transaction log stores (to read and write physical log files and checkpoints).

LogStore is created using LogStoreProvider based on spark.delta.logStore.class configuration property.

Contract

invalidateCache

invalidateCache(): Unit

Used when:

isPartialWriteVisible

isPartialWriteVisible(
  path: Path): Boolean // (1)!
isPartialWriteVisible(
  path: Path,
  hadoopConf: Configuration): Boolean
  1. deprecated

Whether a partial write is visible when writing to path

Default: true

Used when:

listFrom

listFrom(
  path: Path): Iterator[FileStatus]
listFrom(
  path: String): Iterator[FileStatus]  

Used when:

read

read(
  path: Path): Seq[String]
read(
  path: String): Seq[String]

Used when:

write

write(
  path: Path,
  actions: Iterator[String],
  overwrite: Boolean = false): Unit
write(
  path: String,
  actions: Iterator[String]): Unit

Used when:

Implementations

Creating LogStore

apply(
  sc: SparkContext): LogStore
apply(
  sparkConf: SparkConf,
  hadoopConf: Configuration): LogStore

apply creates a LogStore.

apply is used when:

createLogStoreWithClassName

createLogStoreWithClassName(
  className: String,
  sparkConf: SparkConf,
  hadoopConf: Configuration): LogStore

createLogStoreWithClassName branches off based on the given className.

createLogStoreWithClassName creates a DelegatingLogStore when the className is the fully-qualified class name of DelegatingLogStore.

Otherwise, createLogStoreWithClassName loads the class and braches off based on the class type.

For io.delta.storage.LogStore, createLogStoreWithClassName creates an instance thereof (with the given Configuration) and wraps it up in a LogStoreAdaptor.

For all other cases, createLogStoreWithClassName creates an instance thereof (with the given SparkConf and Configuration).

createLogStoreWithClassName is used when:

logStoreSchemeConfKey

logStoreSchemeConfKey(
  scheme: String): String

logStoreSchemeConfKey simply returns the following text for the given scheme:

spark.delta.logStore.[scheme].impl

logStoreSchemeConfKey is used when: