Skip to content

SparkListenerInterface

SparkListenerInterface is an abstraction of event listeners (that SparkListenerBus notifies about scheduling events).

SparkListenerInterface is a way to intercept scheduling events from the Spark Scheduler that are emitted over the course of execution of a Spark application.

SparkListenerInterface is used heavily to manage communication between internal components in the distributed environment for a Spark application (e.g. web UI, event persistence for History Server, dynamic allocation of executors, keeping track of executors).

SparkListenerInterface can be registered in a Spark application using SparkContext.addSparkListener method or spark.extraListeners configuration property.

Tip

Enable INFO logging level for org.apache.spark.SparkContext logger to see what and when custom Spark listeners are registered.

onApplicationEnd

onApplicationEnd(
  applicationEnd: SparkListenerApplicationEnd): Unit

Used when:

onApplicationStart

onApplicationStart(
  applicationStart: SparkListenerApplicationStart): Unit

Used when:

onBlockManagerAdded

onBlockManagerAdded(
  blockManagerAdded: SparkListenerBlockManagerAdded): Unit

Used when:

onBlockManagerRemoved

onBlockManagerRemoved(
  blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit

Used when:

onBlockUpdated

onBlockUpdated(
  blockUpdated: SparkListenerBlockUpdated): Unit

Used when:

onEnvironmentUpdate

onEnvironmentUpdate(
  environmentUpdate: SparkListenerEnvironmentUpdate): Unit

Used when:

onExecutorAdded

onExecutorAdded(
  executorAdded: SparkListenerExecutorAdded): Unit

Used when:

onExecutorBlacklisted

onExecutorBlacklisted(
  executorBlacklisted: SparkListenerExecutorBlacklisted): Unit

Used when:

onExecutorBlacklistedForStage

onExecutorBlacklistedForStage(
  executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit

Used when:

onExecutorMetricsUpdate

onExecutorMetricsUpdate(
  executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit

Used when:

onExecutorRemoved

onExecutorRemoved(
  executorRemoved: SparkListenerExecutorRemoved): Unit

Used when:

onExecutorUnblacklisted

onExecutorUnblacklisted(
  executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit

Used when:

onJobEnd

onJobEnd(
  jobEnd: SparkListenerJobEnd): Unit

Used when:

onJobStart

onJobStart(
  jobStart: SparkListenerJobStart): Unit

Used when:

onNodeBlacklisted

onNodeBlacklisted(
  nodeBlacklisted: SparkListenerNodeBlacklisted): Unit

Used when:

onNodeBlacklistedForStage

onNodeBlacklistedForStage(
  nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit

Used when:

onNodeUnblacklisted

onNodeUnblacklisted(
  nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit

Used when:

onOtherEvent

onOtherEvent(
  event: SparkListenerEvent): Unit

Used when:

onSpeculativeTaskSubmitted

onSpeculativeTaskSubmitted(
  speculativeTask: SparkListenerSpeculativeTaskSubmitted): Unit

Used when:

onStageCompleted

onStageCompleted(
  stageCompleted: SparkListenerStageCompleted): Unit

Used when:

onStageExecutorMetrics

onStageExecutorMetrics(
  executorMetrics: SparkListenerStageExecutorMetrics): Unit

Used when:

onStageSubmitted

onStageSubmitted(
  stageSubmitted: SparkListenerStageSubmitted): Unit

Used when:

onTaskEnd

onTaskEnd(
  taskEnd: SparkListenerTaskEnd): Unit

Used when:

onTaskGettingResult

onTaskGettingResult(
  taskGettingResult: SparkListenerTaskGettingResult): Unit

Used when:

onTaskStart

onTaskStart(
  taskStart: SparkListenerTaskStart): Unit

Used when:

onUnpersistRDD

onUnpersistRDD(
  unpersistRDD: SparkListenerUnpersistRDD): Unit

Used when:

Implementations


Last update: 2020-11-27
Back to top