AppStatusStore¶

AppStatusStore stores the state of a Spark application in a data store (listening to state changes using AppStatusListener).

Creating Instance¶

AppStatusStore takes the following to be created:

• KVStore
• AppStatusListener

AppStatusStore is created using createLiveStore utility.

Creating In-Memory Store for Live Spark Application¶

createLiveStore(
  conf: SparkConf,
  appStatusSource: Option[AppStatusSource] = None): AppStatusStore

createLiveStore creates an ElementTrackingStore (with InMemoryStore and the SparkConf).

createLiveStore creates an AppStatusListener (with the ElementTrackingStore, live flag on and the AppStatusSource).

In the end, creates an AppStatusStore (with the ElementTrackingStore and AppStatusListener).

createLiveStore is used when:

• SparkContext is created

Accessing AppStatusStore¶

AppStatusStore is available using SparkContext.

SparkStatusTracker¶

AppStatusStore is used to create SparkStatusTracker.

SparkUI¶

AppStatusStore is used to create SparkUI.

RDDs¶

rddList(
  cachedOnly: Boolean = true): Seq[v1.RDDStorageInfo]

rddList requests the KVStore for (a view over) RDDStorageInfos (cached or not based on the given cachedOnly flag).

rddList is used when:

• AbstractApplicationResource is requested for the RDDs
• StageTableBase is created (and renders a stage table for AllStagesPage, JobPage and PoolPage)
• StoragePage is requested to render

Streaming Blocks¶

streamBlocksList(): Seq[StreamBlockData]

streamBlocksList requests the KVStore for (a view over) StreamBlockDatas.

streamBlocksList is used when:

• StoragePage is requested to render

Stages¶

stageList(
  statuses: JList[v1.StageStatus]): Seq[v1.StageData]

stageList requests the KVStore for (a view over) StageDatas.

stageList is used when:

• SparkStatusTracker is requested for active stage IDs
• StagesResource is requested for stages
• AllStagesPage is requested to render

Jobs¶

jobsList(
  statuses: JList[JobExecutionStatus]): Seq[v1.JobData]

jobsList requests the KVStore for (a view over) JobDatas.

jobsList is used when:

• SparkStatusTracker is requested for getJobIdsForGroup and getActiveJobIds
• AbstractApplicationResource is requested for jobs
• AllJobsPage is requested to render

Executors¶

executorList(
  activeOnly: Boolean): Seq[v1.ExecutorSummary]

executorList requests the KVStore for (a view over) ExecutorSummarys.

executorList is used when:

• FIXME

Application Summary¶

appSummary(): AppSummary

appSummary requests the KVStore to read the AppSummary.

appSummary is used when:

• AllJobsPage is requested to render
• AllStagesPage is requested to render