Skip to content

AppStatusStore

AppStatusStore stores the state of a Spark application in a data store (listening to state changes using AppStatusListener).

Creating Instance

AppStatusStore takes the following to be created:

AppStatusStore is created using createLiveStore utility.

AppStatusStore in Spark Application

Creating In-Memory Store for Live Spark Application

createLiveStore(
  conf: SparkConf,
  appStatusSource: Option[AppStatusSource] = None): AppStatusStore

createLiveStore creates an ElementTrackingStore (with InMemoryStore and the SparkConf).

createLiveStore creates an AppStatusListener (with the ElementTrackingStore, live flag on and the AppStatusSource).

In the end, creates an AppStatusStore (with the ElementTrackingStore and AppStatusListener).

createLiveStore is used when:

Accessing AppStatusStore

AppStatusStore is available using SparkContext.

SparkStatusTracker

AppStatusStore is used to create SparkStatusTracker.

SparkUI

AppStatusStore is used to create SparkUI.

RDDs

rddList(
  cachedOnly: Boolean = true): Seq[v1.RDDStorageInfo]

rddList requests the KVStore for (a view over) RDDStorageInfos (cached or not based on the given cachedOnly flag).

rddList is used when:

Streaming Blocks

streamBlocksList(): Seq[StreamBlockData]

streamBlocksList requests the KVStore for (a view over) StreamBlockDatas.

streamBlocksList is used when:

  • StoragePage is requested to render

Stages

stageList(
  statuses: JList[v1.StageStatus]): Seq[v1.StageData]

stageList requests the KVStore for (a view over) StageDatas.

stageList is used when:

Jobs

jobsList(
  statuses: JList[JobExecutionStatus]): Seq[v1.JobData]

jobsList requests the KVStore for (a view over) JobDatas.

jobsList is used when:

Executors

executorList(
  activeOnly: Boolean): Seq[v1.ExecutorSummary]

executorList requests the KVStore for (a view over) ExecutorSummarys.

executorList is used when:

  • FIXME

Application Summary

appSummary(): AppSummary

appSummary requests the KVStore to read the AppSummary.

appSummary is used when:

  • AllJobsPage is requested to render
  • AllStagesPage is requested to render

Last update: 2021-06-10