Checkpoints¶
Checkpoints
is an <
[[contract]] .Checkpoints Contract (Abstract Methods Only) [cols="30m,70",options="header",width="100%"] |=== | Method | Description
| logPath a| [[logPath]]
[source, scala]¶
logPath: Path¶
Used when...FIXME
| dataPath a| [[dataPath]]
[source, scala]¶
dataPath: Path¶
Used when...FIXME
| snapshot a| [[snapshot]]
[source, scala]¶
snapshot: Snapshot¶
Used when...FIXME
| store a| [[store]]
[source, scala]¶
store: LogStore¶
Used when...FIXME
| metadata a| [[metadata]]
[source, scala]¶
metadata: Metadata¶
<
Used when...FIXME
| doLogCleanup a| [[doLogCleanup]]
[source, scala]¶
doLogCleanup(): Unit¶
Used when...FIXME
|===
[[implementations]][[self]] NOTE: <Checkpoints
in Delta Lake.
== [[LAST_CHECKPOINT]][[_last_checkpoint]] _last_checkpoint Metadata File
Checkpoints
uses _last_checkpoint metadata file (under the <
-
<
> -
<
>
== [[checkpoint]] Checkpointing -- checkpoint
Method
[source, scala]¶
checkpoint(): Unit checkpoint( snapshotToCheckpoint: Snapshot): CheckpointMetaData
checkpoint
...FIXME
NOTE: checkpoint
is used when...FIXME
Loading Latest Checkpoint Metadata¶
lastCheckpoint: Option[CheckpointMetaData]
lastCheckpoint
simply loadMetadataFromFile (allowing for 3 retries).
lastCheckpoint
is used when:
SnapshotManagement
is requested to load the latest snapshotMetadataCleanup
is requested to listExpiredDeltaLogs
loadMetadataFromFile Helper Method¶
loadMetadataFromFile(
tries: Int): Option[CheckpointMetaData]
loadMetadataFromFile
loads the _last_checkpoint file (in JSON format) and converts it to CheckpointMetaData
(with a version, size and parts).
loadMetadataFromFile
uses the LogStore to read the _last_checkpoint file.
In case the _last_checkpoint file is corrupted, loadMetadataFromFile
...FIXME
== [[manuallyLoadCheckpoint]] manuallyLoadCheckpoint
Method
[source, scala]¶
manuallyLoadCheckpoint(cv: CheckpointInstance): CheckpointMetaData¶
manuallyLoadCheckpoint
...FIXME
NOTE: manuallyLoadCheckpoint
is used when...FIXME
== [[findLastCompleteCheckpoint]] findLastCompleteCheckpoint
Method
[source, scala]¶
findLastCompleteCheckpoint(cv: CheckpointInstance): Option[CheckpointInstance]¶
findLastCompleteCheckpoint
...FIXME
NOTE: findLastCompleteCheckpoint
is used when...FIXME
== [[getLatestCompleteCheckpointFromList]] getLatestCompleteCheckpointFromList
Method
[source, scala]¶
getLatestCompleteCheckpointFromList( instances: Array[CheckpointInstance], notLaterThan: CheckpointInstance): Option[CheckpointInstance]
getLatestCompleteCheckpointFromList
...FIXME
NOTE: getLatestCompleteCheckpointFromList
is used when...FIXME
== [[writeCheckpoint]] writeCheckpoint
Utility
[source, scala]¶
writeCheckpoint( spark: SparkSession, deltaLog: DeltaLog, snapshot: Snapshot): CheckpointMetaData
writeCheckpoint
...FIXME
NOTE: writeCheckpoint
is used when Checkpoints
is requested to <