StreamingQueryListenerBus¶

StreamingQueryListenerBus is an event bus for dispatching streaming events (of active streaming queries) to StreamingQueryListeners.

Creating Instance¶

StreamingQueryListenerBus takes the following to be created:

• LiveListenerBus (Spark Core)

When created, StreamingQueryListenerBus registers itself with the LiveListenerBus to streams event queue.

StreamingQueryListenerBus is created for StreamingQueryManager (once per SparkSession).

SparkListener¶

StreamingQueryListenerBus is an event listener (SparkListener) and registers itself with the LiveListenerBus to intercept QueryStartedEvents.

Run IDs of Active Streaming Queries¶

activeQueryRunIds: Set[UUID]

activeQueryRunIds is an internal registry of run IDs of active streaming queries in the SparkSession.

• A runId is added when StreamingQueryListenerBus is requested to post a QueryStartedEvent

• A runId is removed when StreamingQueryListenerBus is requested to post a QueryTerminatedEvent

activeQueryRunIds is used internally to dispatch a streaming event to a StreamingQueryListener (so the events gets sent out to streaming queries in the SparkSession).

Posting Streaming Event to LiveListenerBus¶

post(
  event: StreamingQueryListener.Event): Unit

post simply posts the input event directly to the LiveListenerBus unless it is a QueryStartedEvent.

For a QueryStartedEvent, post adds the runId (of the streaming query that has been started) to the activeQueryRunIds internal registry first, posts the event to the LiveListenerBus and then postToAll.

post is used when StreamingQueryManager is requested to post a streaming event.

Notifying Listener about Event¶

doPostEvent(
  listener: StreamingQueryListener,
  event: StreamingQueryListener.Event): Unit

doPostEvent is part of the ListenerBus (Spark Core) abstraction.

doPostEvent branches per the type of StreamingQueryListener.Event:

• For a QueryStartedEvent, requests the StreamingQueryListener to onQueryStarted

• For a QueryProgressEvent, requests the StreamingQueryListener to onQueryProgress

• For a QueryTerminatedEvent, requests the StreamingQueryListener to onQueryTerminated

For any other event, doPostEvent simply does nothing (swallows it).

Posting Event To All Listeners¶

postToAll(
  event: Event): Unit

postToAll is part of the ListenerBus (Spark Core) abstraction.

postToAll first requests the parent ListenerBus to post the event to all registered listeners.

For a QueryTerminatedEvent, postToAll simply removes the runId (of the streaming query that has been terminated) from the activeQueryRunIds internal registry.