SparkListenerBus — Internal Contract for Spark Event Buses

SparkListenerBus is a private[spark] ListenerBus for SparkListenerInterface listeners that process SparkListenerEvent events.

SparkListenerBus comes with a custom doPostEvent method that simply relays SparkListenerEvent events to appropriate SparkListenerInterface methods.

There are two custom SparkListenerBus listeners: LiveListenerBus and ReplayListenerBus.
Table 1. SparkListenerEvent to SparkListenerInterface’s Method "mapping"
SparkListenerEvent SparkListenerInterface’s Method

SparkListenerStageSubmitted

onStageSubmitted

SparkListenerStageCompleted

onStageCompleted

SparkListenerJobStart

onJobStart

SparkListenerJobEnd

onJobEnd

SparkListenerJobEnd

onJobEnd

SparkListenerTaskStart

onTaskStart

SparkListenerTaskGettingResult

onTaskGettingResult

SparkListenerTaskEnd

onTaskEnd

SparkListenerEnvironmentUpdate

onEnvironmentUpdate

SparkListenerBlockManagerAdded

onBlockManagerAdded

SparkListenerBlockManagerRemoved

onBlockManagerRemoved

SparkListenerUnpersistRDD

onUnpersistRDD

SparkListenerApplicationStart

onApplicationStart

SparkListenerApplicationEnd

onApplicationEnd

SparkListenerExecutorMetricsUpdate

onExecutorMetricsUpdate

SparkListenerExecutorAdded

onExecutorAdded

SparkListenerExecutorRemoved

onExecutorRemoved

SparkListenerBlockUpdated

onBlockUpdated

SparkListenerLogStart

event ignored

other event types

onOtherEvent

ListenerBus Event Bus Contract

ListenerBus[L <: AnyRef, E]

ListenerBus is an event bus that post events (of type E) to all registered listeners (of type L).

It manages listeners of type L, i.e. it can add to and remove listeners from an internal listeners collection.

addListener(listener: L): Unit
removeListener(listener: L): Unit

It can post events of type E to all registered listeners (using postToAll method). It simply iterates over the internal listeners collection and executes the abstract doPostEvent method.

doPostEvent(listener: L, event: E): Unit
doPostEvent is provided by more specialized ListenerBus event buses.

In case of exception while posting an event to a listener you should see the following ERROR message in the logs and the exception.

ERROR Listener [listener] threw an exception
There are three custom ListenerBus listeners: SparkListenerBus, StreamingQueryListenerBus, and StreamingListenerBus.

Enable ERROR logging level for org.apache.spark.util.ListenerBus logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.util.ListenerBus=ERROR

Refer to Logging.