AppStatusListener¶

AppStatusListener is a SparkListener that writes application state information to a data store.

Event Handlers¶

Creating Instance¶

AppStatusListener takes the following to be created:

• ElementTrackingStore
• SparkConf
• live flag
• AppStatusSource (default: None)
• Last Update Time (default: None)

AppStatusListener is created when:

• AppStatusStore is requested for a in-memory store for a running Spark application (with the live flag enabled)
• FsHistoryProvider is requested to rebuildAppStore (with the live flag disabled)

ElementTrackingStore¶

AppStatusListener is given an ElementTrackingStore when created.

AppStatusListener registers triggers to clean up state in the store:

• cleanupExecutors
• cleanupJobs
• cleanupStages

ElementTrackingStore is used to write and...FIXME

live Flag¶

AppStatusListener is given a live flag when created.

live flag indicates whether AppStatusListener is created for the following:

• true when created for a active (live) Spark application (for AppStatusStore)
• false when created for Spark History Server (for FsHistoryProvider)

Updating ElementTrackingStore for Active Spark Application¶

liveUpdate(
  entity: LiveEntity,
  now: Long): Unit

liveUpdate update the ElementTrackingStore when the live flag is enabled.

Updating ElementTrackingStore¶

update(
  entity: LiveEntity,
  now: Long,
  last: Boolean = false): Unit

update requests the given LiveEntity to write (with the ElementTrackingStore and checkTriggers flag being the given last flag).

getOrCreateExecutor¶

getOrCreateExecutor(
  executorId: String,
  addTime: Long): LiveExecutor

getOrCreateExecutor...FIXME

getOrCreateExecutor is used when:

• AppStatusListener is requested to onExecutorAdded and onBlockManagerAdded

getOrCreateStage¶

getOrCreateStage(
  info: StageInfo): LiveStage

getOrCreateStage...FIXME

getOrCreateStage is used when:

• AppStatusListener is requested to onJobStart and onStageSubmitted