AppStatusListener

AppStatusListener is a SparkListener.

Creating Instance

AppStatusListener takes the following to be created:

AppStatusListener is created when:

live Flag

AppStatusListener is given a flag that indicates whether it is created for a live Spark application (for AppStatusStore) or when replaying Spark applications for Spark History Server (for FsHistoryProvider).

Event Handlers

Event Handler

SparkListenerApplicationStart

onApplicationStart

SparkListenerApplicationEnd

onApplicationEnd

SparkListenerBlockManagerAdded

onBlockManagerAdded

SparkListenerBlockManagerRemoved

onBlockManagerRemoved

SparkListenerBlockUpdated

onBlockUpdated

SparkListenerEnvironmentUpdate

onEnvironmentUpdate

SparkListenerEvent

onOtherEvent

SparkListenerExecutorAdded

onExecutorAdded

SparkListenerExecutorBlacklisted

onExecutorBlacklisted

SparkListenerExecutorMetricsUpdate

onExecutorMetricsUpdate

SparkListenerExecutorRemoved

onExecutorRemoved

SparkListenerExecutorUnblacklisted

onExecutorUnblacklisted

SparkListenerJobStart

onJobStart

SparkListenerJobEnd

onJobEnd

SparkListenerNodeBlacklisted

onNodeBlacklisted

SparkListenerNodeUnblacklisted

onNodeUnblacklisted

SparkListenerStageCompleted

onStageCompleted

SparkListenerStageSubmitted

onStageSubmitted

SparkListenerTaskEnd

onTaskEnd

SparkListenerTaskGettingResult

onTaskGettingResult

SparkListenerTaskStart

onTaskStart

SparkListenerUnpersistRDD

onUnpersistRDD

onStageSubmitted Method

onStageSubmitted(event: SparkListenerStageSubmitted): Unit
onStageSubmitted is part of SparkListener Contract to…​FIXME.

onStageSubmitted…​FIXME

update Internal Method

update(entity: LiveEntity, now: Long, last: Boolean = false): Unit

update simply requests the LiveEntity to write (with the ElementTrackingStore as the store and the last flag as checkTriggers flag).

update is used in event handlers (i.e. onApplicationStart, onExecutorRemoved, onJobEnd, onStageSubmitted, onTaskEnd, onStageCompleted), liveUpdate, maybeUpdate, flush and updateRDDBlock.

flush Internal Method

flush(): Unit

flush…​FIXME

flush is used when…​FIXME

maybeUpdate Internal Method

maybeUpdate(entity: LiveEntity, now: Long): Unit

maybeUpdate…​FIXME

maybeUpdate is used when…​FIXME

liveUpdate Internal Method

liveUpdate(entity: LiveEntity, now: Long): Unit

liveUpdate…​FIXME

liveUpdate is used when…​FIXME

updateStreamBlock Internal Method

updateStreamBlock(event: SparkListenerBlockUpdated, stream: StreamBlockId): Unit

updateStreamBlock…​FIXME

updateStreamBlock is used exclusively when AppStatusListener is requested to handle a SparkListenerBlockUpdated event (for a StreamBlockId).

Intercepting SparkListenerBlockUpdated Events — onBlockUpdated Handler Method

onBlockUpdated(event: SparkListenerBlockUpdated): Unit
onBlockUpdated is part of SparkListener Contract to…​FIXME.

onBlockUpdated simply dispatches to the following event-specific handlers (per BlockId type):

updateRDDBlock Internal Method

updateRDDBlock(
  event: SparkListenerBlockUpdated,
  block: RDDBlockId): Unit

updateRDDBlock…​FIXME

updateRDDBlock is used exclusively when AppStatusListener is requested to handle a SparkListenerBlockUpdated event (for a RDDBlockId).

updateBroadcastBlock Internal Method

updateBroadcastBlock(
  event: SparkListenerBlockUpdated,
  broadcast: BroadcastBlockId): Unit

updateBroadcastBlock…​FIXME

updateBroadcastBlock is used…​FIXME

Internal Properties

Name Description

appInfo

v1.ApplicationInfo

appSummary

AppSummary

liveUpdatePeriodNs

coresPerTask

Default: 1

liveRDDs

LiveRDDs by RDD ID

liveStages

LiveStages by (Int, Int)

liveTasks

LiveTask by task ID

liveJobs

LiveJob by job ID

liveExecutors

LiveExecutor by executor ID

pools

SchedulerPool by FIXME

activeExecutorCount

Number of active executors