Skip to content

InMemoryLogReplay — Delta Log Replay

InMemoryLogReplay is used at the very last phase of <> (of a <>).

InMemoryLogReplay is <> for every partition of the <> dataset (Dataset[SingleAction]) that is based on the <> configuration property (default: 50).

The lifecycle of InMemoryLogReplay is as follows:

. <> (with <>)

. <> (with all <> of a partition)

. <>

== [[creating-instance]] Creating InMemoryLogReplay Instance

InMemoryLogReplay takes the following to be created:

  • [[minFileRetentionTimestamp]] minFileRetentionTimestamp (that is exactly <>)

InMemoryLogReplay initializes the <>.

== [[append]] Appending Actions -- append Method

[source, scala]

append( version: Long, actions: Iterator[Action]): Unit


append sets the <> as the given version.

append adds <> to respective registries:

  • Every <> is registered in the <> by <>

  • <> is registered as the <>

  • <> is registered as the <>

  • Every <> is registered as follows: ** Added to <> by pathAsUri (with dataChange flag turned off) ** Removed from <> by pathAsUri

  • Every <> is registered as follows: ** Removed from <> by pathAsUri ** Added to <> by pathAsUri (with dataChange flag turned off)

  • <> are ignored

append throws an AssertionError when the <> is neither -1 (the default) nor one before the given version:

Attempted to replay version [version], but state is at [currentVersion]

NOTE: append is used when Snapshot is created (and initializes the <> for the <>).

== [[checkpoint]] Current State Of Delta Table -- checkpoint Method

[source, scala]

checkpoint: Iterator[Action]

checkpoint simply builds a sequence (Iterator[Action]) of the following (in that order):

  • <> if defined (non-null)

  • <> if defined (non-null)

  • <>

  • <> and <> (after the <>) sorted by <> (lexicographically)

NOTE: checkpoint is used when Snapshot is created (and initializes the <> for the <>).

== [[getTombstones]] getTombstones Internal Method

[source, scala]

getTombstones: Iterable[FileAction]

getTombstones returns <> (from the <>) with their delTimestamp after the <>.

NOTE: getTombstones is used when InMemoryLogReplay is requested to <>.

== [[internal-properties]] Internal Properties

[cols="30m,70",options="header",width="100%"] |=== | Name | Description

| currentProtocolVersion a| [[currentProtocolVersion]] <> (default: null)

Used when...FIXME

| currentVersion a| [[currentVersion]] Version (default: -1)

Used when...FIXME

| currentMetaData a| [[currentMetaData]] <> (default: null)

Used when...FIXME

| transactions a| [[transactions]] <> per ID (HashMap[String, SetTransaction])

Used when...FIXME

| activeFiles a| [[activeFiles]] <> per URI (HashMap[URI, AddFile])

Used when...FIXME

| tombstones a| [[tombstones]] <> per URI (HashMap[URI, RemoveFile])

Used when...FIXME

|===


Last update: 2020-09-29