ExecutionListenerManager¶

ExecutionListenerManager is a frontend (facade) of ExecutionListenerBus to manage QueryExecutionListeners (in a SparkSession).

Creating Instance¶

ExecutionListenerManager takes the following to be created:

• SparkSession
• SQLConf
• loadExtensions flag

ExecutionListenerManager is created when:

• BaseSessionStateBuilder is requested for the session ExecutionListenerManager (while SessionState is built)
• ExecutionListenerManager is requested to clone

ExecutionListenerBus¶

ExecutionListenerManager creates an ExecutionListenerBus when created with the following:

• This ExecutionListenerManager
• SparkSession

The ExecutionListenerBus is used for the following:

• Register a QueryExecutionListener
• Unregister a QueryExecutionListener
• Unregister all QueryExecutionListeners
• clone

Accessing ExecutionListenerManager¶

ExecutionListenerManager is available as SparkSession.listenerManager (and SessionState.listenerManager).

scala> :type spark.listenerManager
org.apache.spark.sql.util.ExecutionListenerManager

scala> :type spark.sessionState.listenerManager
org.apache.spark.sql.util.ExecutionListenerManager

spark.sql.queryExecutionListeners¶

ExecutionListenerManager is given loadExtensions flag when created.

When enabled, ExecutionListenerManager registers the QueryExecutionListeners that are configured using the spark.sql.queryExecutionListeners configuration property.

Removing All QueryExecutionListeners¶

clear(): Unit

Registering QueryExecutionListener¶

register(
  listener: QueryExecutionListener): Unit

register requests the ExecutionListenerBus to register the given QueryExecutionListener.

register is used when:

• Observation is requested to register
• ExecutionListenerManager is created and cloned

De-registering QueryExecutionListener¶

unregister(
  listener: QueryExecutionListener): Unit

Logging¶

Enable ALL logging level for org.apache.spark.sql.util.ExecutionListenerManager logger to see what happens inside.

Add the following line to conf/log4j2.properties:

log4j.logger.org.apache.spark.sql.util.ExecutionListenerManager=ALL

Refer to Logging.