SparkListenerInterface¶

SparkListenerInterface is an abstraction of event listeners (that SparkListenerBus notifies about scheduling events).

SparkListenerInterface is a way to intercept scheduling events from the Spark Scheduler that are emitted over the course of execution of a Spark application.

SparkListenerInterface is used heavily to manage communication between internal components in the distributed environment for a Spark application (e.g. web UI, event persistence for History Server, dynamic allocation of executors, keeping track of executors).

SparkListenerInterface can be registered in a Spark application using SparkContext.addSparkListener method or spark.extraListeners configuration property.

Tip

Enable INFO logging level for org.apache.spark.SparkContext logger to see what and when custom Spark listeners are registered.

onApplicationEnd¶

onApplicationEnd(
  applicationEnd: SparkListenerApplicationEnd): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerApplicationEnd event

onApplicationStart¶

onApplicationStart(
  applicationStart: SparkListenerApplicationStart): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerApplicationStart event

onBlockManagerAdded¶

onBlockManagerAdded(
  blockManagerAdded: SparkListenerBlockManagerAdded): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerBlockManagerAdded event

onBlockManagerRemoved¶

onBlockManagerRemoved(
  blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerBlockManagerRemoved event

onBlockUpdated¶

onBlockUpdated(
  blockUpdated: SparkListenerBlockUpdated): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerBlockUpdated event

onEnvironmentUpdate¶

onEnvironmentUpdate(
  environmentUpdate: SparkListenerEnvironmentUpdate): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerEnvironmentUpdate event

onExecutorAdded¶

onExecutorAdded(
  executorAdded: SparkListenerExecutorAdded): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerExecutorAdded event

onExecutorBlacklisted¶

onExecutorBlacklisted(
  executorBlacklisted: SparkListenerExecutorBlacklisted): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerExecutorBlacklisted event

onExecutorBlacklistedForStage¶

onExecutorBlacklistedForStage(
  executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerExecutorBlacklistedForStage event

onExecutorMetricsUpdate¶

onExecutorMetricsUpdate(
  executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerExecutorMetricsUpdate event

onExecutorRemoved¶

onExecutorRemoved(
  executorRemoved: SparkListenerExecutorRemoved): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerExecutorRemoved event

onExecutorUnblacklisted¶

onExecutorUnblacklisted(
  executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerExecutorUnblacklisted event

onJobEnd¶

onJobEnd(
  jobEnd: SparkListenerJobEnd): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerJobEnd event

onJobStart¶

onJobStart(
  jobStart: SparkListenerJobStart): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerJobStart event

onNodeBlacklisted¶

onNodeBlacklisted(
  nodeBlacklisted: SparkListenerNodeBlacklisted): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerNodeBlacklisted event

onNodeBlacklistedForStage¶

onNodeBlacklistedForStage(
  nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerNodeBlacklistedForStage event

onNodeUnblacklisted¶

onNodeUnblacklisted(
  nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerNodeUnblacklisted event

onOtherEvent¶

onOtherEvent(
  event: SparkListenerEvent): Unit

Used when:

SparkListenerBus is requested to post a custom SparkListenerEvent

onSpeculativeTaskSubmitted¶

onSpeculativeTaskSubmitted(
  speculativeTask: SparkListenerSpeculativeTaskSubmitted): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerSpeculativeTaskSubmitted event

onStageCompleted¶

onStageCompleted(
  stageCompleted: SparkListenerStageCompleted): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerStageCompleted event

onStageExecutorMetrics¶

onStageExecutorMetrics(
  executorMetrics: SparkListenerStageExecutorMetrics): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerStageExecutorMetrics event

onStageSubmitted¶

onStageSubmitted(
  stageSubmitted: SparkListenerStageSubmitted): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerStageSubmitted event

onTaskEnd¶

onTaskEnd(
  taskEnd: SparkListenerTaskEnd): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerTaskEnd event

onTaskGettingResult¶

onTaskGettingResult(
  taskGettingResult: SparkListenerTaskGettingResult): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerTaskGettingResult event

onTaskStart¶

onTaskStart(
  taskStart: SparkListenerTaskStart): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerTaskStart event

onUnpersistRDD¶

onUnpersistRDD(
  unpersistRDD: SparkListenerUnpersistRDD): Unit

Used when:

SparkListenerBus is requested to post a SparkListenerUnpersistRDD event

Implementations¶

EventFilterBuilder
SparkFirehoseListener
SparkListener