LiveListenerBus is an event bus to announce application-wide events in a Spark application. It asynchronously passes <
.LiveListenerBus, SparkListenerEvents, and Senders image::spark-sparklistener-event-senders.png[align="center"]
LiveListenerBus is a single-JVM spark-SparkListenerBus.md[SparkListenerBus] that uses <
NOTE: The event queue is http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/LinkedBlockingQueue.html[java.util.concurrent.LinkedBlockingQueue] with capacity of 10000
== [[creating-instance]] Creating Instance
LiveListenerBus takes the following to be created:
- [[conf]] ROOT:SparkConf.md
LiveListenerBus is created (and <
== [[start]] Starting LiveListenerBus
start( sc: SparkContext): Unit
start starts <
Internally, it saves the input
SparkContext for later use and starts <
started is disabled).
If however LiveListenerBus has already been started, a
IllegalStateException is thrown:
[name] already started!
== [[post]] Posting SparkListenerEvent Event
post( event: SparkListenerEvent): Unit
post puts the input
event onto the internal
eventQueue queue and releases the internal
eventLock semaphore. If the event placement was not successful (and it could happen since it is tapped at 10000 events) <
The event publishing is only possible when
stopped flag has been enabled.
CAUTION: FIXME Who's enabling the
stopped flag and when/why?
If LiveListenerBus has been stopped, the following ERROR appears in the logs:
[name] has already stopped! Dropping event [event]
== [[onDropEvent]] Event Dropped Callback
onDropEvent( event: SparkListenerEvent): Unit
onDropEvent is called when no further events can be added to the internal
eventQueue queue (while <
It simply prints out the following ERROR message to the logs and ensures that it happens only once.
ERROR Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
NOTE: It uses the internal
logDroppedEvent atomic variable to track the state.
== [[stop]] Stopping LiveListenerBus
stop releases the internal
eventLock semaphore and waits until <
eventQueue gives nothing).
It checks that
started flag is enabled (i.e.
true) and throws a
Attempted to stop [name] that has not yet started!
stopped flag is enabled.
== [[listenerThread]] listenerThread for Event Polling
LiveListenerBus uses ROOT:spark-SparkListenerBus.md single-daemon thread that ensures that the polling events from the event queue is only after <
CAUTION: FIXME There is some logic around no events in the queue.
== [[addToStatusQueue]] Registering SparkListenerInterface with Application Status Queue
addToStatusQueue( listener: SparkListenerInterface): Unit
addToStatusQueue simply <
addToStatusQueue is used when...FIXME
== [[addToQueue]] Registering SparkListenerInterface with Queue
addToQueue( listener: SparkListenerInterface, queue: String): Unit
addToQueue is used when...FIXME