Checkpoints

Checkpoints is an abstraction of DeltaLogs that can checkpoint the current state of a delta table (represented by the DeltaLog).

Table 1. Checkpoints Contract (Abstract Methods Only)
Method Description

logPath

logPath: Path

Used when…​FIXME

dataPath

dataPath: Path

Used when…​FIXME

snapshot

snapshot: Snapshot

Used when…​FIXME

store

store: LogStore

Used when…​FIXME

metadata

metadata: Metadata

Metadata of (the current state of) the delta table

Used when…​FIXME

doLogCleanup

doLogCleanup(): Unit

Used when…​FIXME

NOTE: DeltaLog is the default and only known Checkpoints in Delta Lake.

_last_checkpoint Metadata File

Checkpoints uses _last_checkpoint metadata file (under the log path) for the following:

Checkpointing — checkpoint Method

checkpoint(): Unit
checkpoint(
  snapshotToCheckpoint: Snapshot): CheckpointMetaData

checkpoint…​FIXME

checkpoint is used when…​FIXME

lastCheckpoint Method

lastCheckpoint: Option[CheckpointMetaData]

lastCheckpoint simply loadMetadataFromFile (with 0 tries).

lastCheckpoint is used when…​FIXME

manuallyLoadCheckpoint Method

manuallyLoadCheckpoint(cv: CheckpointInstance): CheckpointMetaData

manuallyLoadCheckpoint…​FIXME

manuallyLoadCheckpoint is used when…​FIXME

findLastCompleteCheckpoint Method

findLastCompleteCheckpoint(cv: CheckpointInstance): Option[CheckpointInstance]

findLastCompleteCheckpoint…​FIXME

findLastCompleteCheckpoint is used when…​FIXME

getLatestCompleteCheckpointFromList Method

getLatestCompleteCheckpointFromList(
  instances: Array[CheckpointInstance],
  notLaterThan: CheckpointInstance): Option[CheckpointInstance]

getLatestCompleteCheckpointFromList…​FIXME

getLatestCompleteCheckpointFromList is used when…​FIXME

writeCheckpoint Utility

writeCheckpoint(
  spark: SparkSession,
  deltaLog: DeltaLog,
  snapshot: Snapshot): CheckpointMetaData

writeCheckpoint…​FIXME

writeCheckpoint is used when Checkpoints is requested to checkpoint.

loadMetadataFromFile Internal Method

loadMetadataFromFile(tries: Int): Option[CheckpointMetaData]

loadMetadataFromFile…​FIXME

loadMetadataFromFile is used when…​FIXME