LiveListenerBus is an event bus to dispatch Spark events to registered SparkListeners.
The event queue is java.util.concurrent.LinkedBlockingQueue with capacity of 10000
LiveListenerBus takes the following to be created:
LiveListenerBus manages AsyncEventQueues.
queues is initialized empty when
LiveListenerBus is created.
queues is used when:
- Registering Listener with Queue
- Posting Event to All Queues
- Deregistering Listener
- Starting LiveListenerBus
LiveListenerBus creates a
LiveListenerBusMetrics when created.
metrics is used to:
- Increment events posted every event posting
- Create a AsyncEventQueue when adding a listener to a queue
start( sc: SparkContext, metricsSystem: MetricsSystem): Unit
start is used when:
Posting Event to All Queues¶
post( event: SparkListenerEvent): Unit
post puts the input
event onto the internal
eventQueue queue and releases the internal
eventLock semaphore. If the event placement was not successful (and it could happen since it is tapped at 10000 events) onDropEvent method is called.
The event publishing is only possible when
stopped flag has been enabled.
post is used when...FIXME
postToQueues( event: SparkListenerEvent): Unit
Event Dropped Callback¶
onDropEvent( event: SparkListenerEvent): Unit
onDropEvent is called when no further events can be added to the internal
eventQueue queue (while posting a SparkListenerEvent event).
It simply prints out the following ERROR message to the logs and ensures that it happens only once.
Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
stop releases the internal
eventLock semaphore and waits until listenerThread dies. It can only happen after all events were posted (and polling
eventQueue gives nothing).
stopped flag is enabled.
listenerThread for Event Polling¶
Registering Listener with Status Queue¶
addToStatusQueue( listener: SparkListenerInterface): Unit
addToStatusQueue is used when:
BarrierCoordinatoris requested to
HiveThriftServer2utility is used to
SharedState(Spark SQL) is requested to create a SQLAppStatusStore
addToSharedQueue( listener: SparkListenerInterface): Unit
addToSharedQueue is used when:
SparkContextis requested to register a SparkListener and register extra SparkListeners
ExecutionListenerBus(Spark Structured Streaming) is created
Registering Listener with executorManagement Queue¶
addToManagementQueue( listener: SparkListenerInterface): Unit
addToManagementQueue is used when:
Registering Listener with eventLog Queue¶
addToEventLogQueue( listener: SparkListenerInterface): Unit
addToEventLogQueue is used when:
SparkContextis created (with event logging enabled)
Registering Listener with Queue¶
addToQueue( listener: SparkListenerInterface, queue: String): Unit
addToQueue finds the queue in the queues internal registry.
addToQueue requests it to add the given listener
If not found,
addToQueue creates a AsyncEventQueue (with the given name, the LiveListenerBusMetrics, and this
LiveListenerBus) and requests it to add the given listener. The
AsyncEventQueue is started and added to the queues internal registry.
addToQueue is used when:
LiveListenerBusis requested to addToSharedQueue, addToManagementQueue, addToStatusQueue, addToEventLogQueue
StreamingQueryListenerBus(Spark Structured Streaming) is created
removeListener( listener: SparkListenerInterface): Unit
removeListener is used when: