CachedDS — Cached Delta State¶

CachedDS is used when StateCache is requested to cacheDS.

When created, CachedDS immediately initializes the cachedDs internal registry that requests the Dataset to generate a RDD[InternalRow] and associates the RDD with the given name:

Delta Table State for Snapshot
Delta Source Snapshot for DeltaSourceSnapshot

The RDD is marked to be persisted using StorageLevel.MEMORY_AND_DISK_SER storage level.

Note

CachedDS is an internal class of StateCache and has access to its internals.

Creating Instance¶

CachedDS takes the following to be created:

Dataset[A]
Name

CachedDS is created when StateCache is requested to cacheDS.

getDS Method¶

getDS: Dataset[A]

getDS...FIXME

getDS is used when:

Snapshot is requested to state
DeltaSourceSnapshot is requested to initialFiles