LogStore

LogStore is an abstraction of log stores that can read and write actions to a directory (among other things).

Table 1. LogStore Contract
Method Description

invalidateCache

invalidateCache(): Unit

Used when…​FIXME

isPartialWriteVisible

isPartialWriteVisible(path: Path): Boolean = true

Used when…​FIXME

listFrom

listFrom(
  path: Path): Iterator[FileStatus]
listFrom(
  path: String): Iterator[FileStatus]

Used when…​FIXME

read

read(path: String): Seq[String]
read(path: Path): Seq[String]

Used when:

write

write(
  path: Path,
  actions: Iterator[String],
  overwrite: Boolean = false): Unit
write(
  path: String,
  actions: Iterator[String]): Unit

Writes the actions out to the given path (with or without overwrite as indicated).

Used when:

Table 2. LogStores (Direct Implementations and Extensions Only)
LogStore Description

HDFSLogStore

HadoopFileSystemLogStore

resolvePathOnPhysicalStorage Method

resolvePathOnPhysicalStorage(path: Path): Path

resolvePathOnPhysicalStorage…​FIXME

resolvePathOnPhysicalStorage is used when…​FIXME

Creating LogStore — apply Utility

apply(
  sc: SparkContext): LogStore
apply(
  sparkConf: SparkConf,
  hadoopConf: Configuration): LogStore

apply…​FIXME

apply is used when: