CachedDS — Cached Delta State¶
CachedDS
is used when StateCache is requested to cacheDS.
When created, CachedDS
immediately initializes the cachedDs
internal registry that requests the Dataset to generate a RDD[InternalRow]
and associates the RDD with the given name:
- Delta Table State for Snapshot
- Delta Source Snapshot for DeltaSourceSnapshot
The RDD is marked to be persisted using StorageLevel.MEMORY_AND_DISK_SER
storage level.
Note
CachedDS
is an internal class of StateCache
and has access to its internals.
Creating Instance¶
CachedDS
takes the following to be created:
-
Dataset[A]
- Name
CachedDS
is created when StateCache
is requested to cacheDS.
getDS Method¶
getDS: Dataset[A]
getDS
...FIXME
getDS
is used when:
Snapshot
is requested to stateDeltaSourceSnapshot
is requested to initialFiles
Last update: 2020-09-29