Skip to content

Checkpoints

Checkpoints is an <> of <> that can <> the current state of a delta table (represented by the <>).

[[contract]] .Checkpoints Contract (Abstract Methods Only) [cols="30m,70",options="header",width="100%"] |=== | Method | Description

| logPath a| [[logPath]]

[source, scala]

logPath: Path

Used when...FIXME

| dataPath a| [[dataPath]]

[source, scala]

dataPath: Path

Used when...FIXME

| snapshot a| [[snapshot]]

[source, scala]

snapshot: Snapshot

Used when...FIXME

| store a| [[store]]

[source, scala]

store: LogStore

Used when...FIXME

| metadata a| [[metadata]]

[source, scala]

metadata: Metadata

<> of (the current state of) the <>

Used when...FIXME

| doLogCleanup a| [[doLogCleanup]]

[source, scala]

doLogCleanup(): Unit

Used when...FIXME

|===

[[implementations]][[self]] NOTE: <> is the default and only known Checkpoints in Delta Lake.

== [[LAST_CHECKPOINT]][[_last_checkpoint]] _last_checkpoint Metadata File

Checkpoints uses _last_checkpoint metadata file (under the <>) for the following:

  • <>

  • <>

== [[checkpoint]] Checkpointing -- checkpoint Method

[source, scala]

checkpoint(): Unit checkpoint( snapshotToCheckpoint: Snapshot): CheckpointMetaData


checkpoint...FIXME

NOTE: checkpoint is used when...FIXME

Loading Latest Checkpoint Metadata

lastCheckpoint: Option[CheckpointMetaData]

lastCheckpoint simply loadMetadataFromFile (allowing for 3 retries).

lastCheckpoint is used when:

loadMetadataFromFile Helper Method

loadMetadataFromFile(
  tries: Int): Option[CheckpointMetaData]

loadMetadataFromFile loads the _last_checkpoint file (in JSON format) and converts it to CheckpointMetaData (with a version, size and parts).

loadMetadataFromFile uses the LogStore to read the _last_checkpoint file.

In case the _last_checkpoint file is corrupted, loadMetadataFromFile...FIXME

== [[manuallyLoadCheckpoint]] manuallyLoadCheckpoint Method

[source, scala]

manuallyLoadCheckpoint(cv: CheckpointInstance): CheckpointMetaData

manuallyLoadCheckpoint...FIXME

NOTE: manuallyLoadCheckpoint is used when...FIXME

== [[findLastCompleteCheckpoint]] findLastCompleteCheckpoint Method

[source, scala]

findLastCompleteCheckpoint(cv: CheckpointInstance): Option[CheckpointInstance]

findLastCompleteCheckpoint...FIXME

NOTE: findLastCompleteCheckpoint is used when...FIXME

== [[getLatestCompleteCheckpointFromList]] getLatestCompleteCheckpointFromList Method

[source, scala]

getLatestCompleteCheckpointFromList( instances: Array[CheckpointInstance], notLaterThan: CheckpointInstance): Option[CheckpointInstance]


getLatestCompleteCheckpointFromList...FIXME

NOTE: getLatestCompleteCheckpointFromList is used when...FIXME

== [[writeCheckpoint]] writeCheckpoint Utility

[source, scala]

writeCheckpoint( spark: SparkSession, deltaLog: DeltaLog, snapshot: Snapshot): CheckpointMetaData


writeCheckpoint...FIXME

NOTE: writeCheckpoint is used when Checkpoints is requested to <>.


Last update: 2020-09-27