StreamingQueryStatusListener¶

StreamingQueryStatusListener is a StreamingQueryListener that intercepts streaming events and writes them out to the ElementTrackingStore (for StreamingQueryTab).

Creating Instance¶

StreamingQueryStatusListener takes the following to be created:

• SparkConf (Spark Core)
• ElementTrackingStore (Spark Core)

StreamingQueryStatusListener is created when:

• StreamingQueryHistoryServerPlugin is requested to createListeners
• SharedState (Spark SQL) is created (with spark.sql.streaming.ui.enabled enabled)

spark.sql.streaming.ui.retainedProgressUpdates¶

StreamingQueryStatusListener uses spark.sql.streaming.ui.retainedProgressUpdates configuration property in onQueryProgress to control when to delete excessive StreamingQueryProgresss (in the ElementTrackingStore).

onQueryStarted¶

onQueryStarted(
  event: StreamingQueryListener.QueryStartedEvent): Unit

onQueryStarted is part of the StreamingQueryListener abstraction.

onQueryStarted writes out a new "start" StreamingQueryData to the ElementTrackingStore.

onQueryProgress¶

onQueryProgress(
  event: StreamingQueryListener.QueryProgressEvent): Unit

onQueryProgress is part of the StreamingQueryListener abstraction.

onQueryProgress...FIXME

onQueryTerminated¶

onQueryTerminated(
  event: StreamingQueryListener.QueryTerminatedEvent): Unit

onQueryTerminated is part of the StreamingQueryListener abstraction.

onQueryTerminated finds the query summary (the StreamingQueryData) for the runId in the ElementTrackingStore.

onQueryTerminated writes out a new "terminate" StreamingQueryData to the ElementTrackingStore.

In the end, onQueryTerminated removes the streaming query (by runId) from the queryToProgress registry.