InMemoryLogReplay — Delta Log Replay

InMemoryLogReplay is used at the very last phase of state reconstruction (of a cached delta state).

InMemoryLogReplay is created for every partition of the stateReconstruction dataset (Dataset[SingleAction]) that is based on the spark.databricks.delta.snapshotPartitions configuration property (default: 50).

The lifecycle of InMemoryLogReplay is as follows:

Creating InMemoryLogReplay Instance

InMemoryLogReplay takes the following to be created:

InMemoryLogReplay initializes the internal properties.

Appending Actions — append Method

append(
  version: Long,
  actions: Iterator[Action]): Unit

append sets the currentVersion as the given version.

append adds actions to respective registries:

append throws an AssertionError when the currentVersion is neither -1 (the default) nor one before the given version:

Attempted to replay version [version], but state is at [currentVersion]
append is used when Snapshot is created (and initializes the stateReconstruction for the cached delta state).

Current State Of Delta Table — checkpoint Method

checkpoint: Iterator[Action]

checkpoint simply builds a sequence (Iterator[Action]) of the following (in that order):

checkpoint is used when Snapshot is created (and initializes the stateReconstruction for the cached delta state).

getTombstones Internal Method

getTombstones: Iterable[FileAction]

getTombstones returns RemoveFiles (from the tombstones) with their delTimestamp after the minFileRetentionTimestamp.

getTombstones is used when InMemoryLogReplay is requested to checkpoint.

Internal Properties

Name Description

currentProtocolVersion

Protocol (default: null)

Used when…​FIXME

currentVersion

Version (default: -1)

Used when…​FIXME

currentMetaData

Metadata (default: null)

Used when…​FIXME

transactions

SetTransactions per ID (HashMap[String, SetTransaction])

Used when…​FIXME

activeFiles

AddFile per URI (HashMap[URI, AddFile])

Used when…​FIXME

tombstones

RemoveFile per URI (HashMap[URI, RemoveFile])

Used when…​FIXME