AppStatusListener¶
AppStatusListener
is a SparkListener.
Creating Instance¶
AppStatusListener
takes the following to be created:
- ElementTrackingStore
- SparkConf
- live flag
-
AppStatusSource
(default:None
) - Last Update Time (default:
None
)
AppStatusListener
is created when:
AppStatusStore
is requested to createLiveStore (with the live flag enabled)FsHistoryProvider
is requested to rebuildAppStore (with the live flag disabled)
live Flag¶
AppStatusListener
is given a live
flag when created.
live
flag indicates whether AppStatusListener
is created for the following:
true
when created for a active (live) Spark application (for AppStatusStore)false
when created for Spark History Server (for FsHistoryProvider)
onJobStart¶
onJobStart(
event: SparkListenerJobStart): Unit
onJobStart
...FIXME
onJobStart
is part of the SparkListener abstraction.
onStageSubmitted¶
onStageSubmitted(
event: SparkListenerStageSubmitted): Unit
onStageSubmitted
is part of the SparkListener abstraction.
onStageSubmitted
getOrCreateStage for the StageInfo (from the given SparkListenerStageSubmitted
event).
onStageSubmitted
...FIXME
In the end, onStageSubmitted
liveUpdate with the stage and now
timestamp.
Updating ElementTrackingStore for Active Spark Application¶
liveUpdate(
entity: LiveEntity,
now: Long): Unit
liveUpdate
update the ElementTrackingStore when the live flag is enabled.
Updating ElementTrackingStore¶
update(
entity: LiveEntity,
now: Long,
last: Boolean = false): Unit
update
requests the given LiveEntity to write (with the ElementTrackingStore and checkTriggers
flag being the given last
flag).
getOrCreateStage¶
getOrCreateStage(
info: StageInfo): LiveStage
getOrCreateStage
...FIXME
getOrCreateStage
is used when AppStatusListener
is requested to onJobStart and onStageSubmitted.