CachedDS — Cached Delta State¶

CachedDS is used when StateCache is requested to cacheDS.

When created, CachedDS immediately initializes the cachedDs internal registry that requests the Dataset to generate a RDD[InternalRow] and associates the RDD with the given name:

• Delta Table State for Snapshot
• Delta Source Snapshot for DeltaSourceSnapshot

The RDD is marked to be persisted using StorageLevel.MEMORY_AND_DISK_SER storage level.

Creating Instance¶

CachedDS takes the following to be created:

• Dataset[A]
• Name

CachedDS is created when StateCache is requested to cacheDS.

getDS Method¶

getDS: Dataset[A]

getDS...FIXME

getDS is used when:

• Snapshot is requested to state
• DeltaSourceSnapshot is requested to initialFiles