AppStatusStore is…​FIXME

spark core AppStatusStore createLiveStore
Figure 1. AppStatusStore in Spark Application

AppStatusStore is created when:

When created for a live Spark application, AppStatusStore creates a AppStatusListener that is later used to get the active stages.

streamBlocksList Method

streamBlocksList(): Seq[StreamBlockData]


streamBlocksList is used when…​FIXME

activeStages Method

activeStages(): Seq[v1.StageData]


activeStages is used when…​FIXME

Creating AppStatusStore Instance

AppStatusStore takes the following when created:

Creating Live Store (AppStatusStore For Active Spark Application) — createLiveStore Factory Method

createLiveStore(conf: SparkConf): AppStatusStore

createLiveStore creates a fully-initialized AppStatusStore.

Internally, createLiveStore creates a ElementTrackingStore (with a new InMemoryStore and the input SparkConf).

createLiveStore creates a AppStatusListener (with the ElementTrackingStore created, the input SparkConf and the live flag enabled).

In the end, createLiveStore creates an AppStatusStore (with the ElementTrackingStore and AppStatusListener just created).

createLiveStore is used exclusively when SparkContext is created.

Closing AppStatusStore — close Method

close(): Unit

close simply requests KVStore to close.

close is used when…​FIXME

rddList Method

rddList(cachedOnly: Boolean = true): Seq[v1.RDDStorageInfo]

rddList requests the KVStore for a view over RDDStorageInfoWrapper entities.

In the end, rddList takes RDDStorageInfos with at least one partition cached (when cachedOnly flag is on) or all RDDStorageInfos (when cachedOnly flag is off).

cachedOnly flag is on and therefore rddList gives RDDs cached only.

rddList is used when:

  • StoragePage is requested to render itself

  • AbstractApplicationResource is requested to handle storage/rdd REST API

  • StagePagedTable is requested to makeDescription