Skip to content

EventLoggingListener

EventLoggingListener is a SparkListener that writes out JSON-encoded events of a Spark application with event logging enabled (based on spark.eventLog.enabled configuration property).

EventLoggingListener supports custom configuration properties.

EventLoggingListener writes out log files to a directory (based on spark.eventLog.dir configuration property).

Creating Instance

EventLoggingListener takes the following to be created:

EventLoggingListener is created when SparkContext is created (with spark.eventLog.enabled enabled).

EventLogFileWriter

logWriter: EventLogFileWriter

EventLoggingListener creates a EventLogFileWriter when created.

Note

All arguments to create an EventLoggingListener are passed to the EventLogFileWriter.

The EventLogFileWriter is started when EventLoggingListener is started.

The EventLogFileWriter is stopped when EventLoggingListener is stopped.

The EventLogFileWriter is requested to writeEvent when EventLoggingListener is requested to start and log an event.

Starting EventLoggingListener

start(): Unit

start requests the EventLogFileWriter to start and initEventLog.

initEventLog

initEventLog(): Unit

initEventLog...FIXME

Logging Event

logEvent(
  event: SparkListenerEvent,
  flushLogger: Boolean = false): Unit

logEvent persists the given SparkListenerEvent in JSON format.

logEvent converts the event to JSON format and requests the EventLogFileWriter to write it out.

Stopping EventLoggingListener

stop(): Unit

stop requests the EventLogFileWriter to stop.

stop is used when SparkContext is requested to stop.

inprogress File Extension

EventLoggingListener uses .inprogress file extension for in-flight event log files of active Spark applications.

Logging

Enable ALL logging level for org.apache.spark.scheduler.EventLoggingListener logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.scheduler.EventLoggingListener=ALL

Refer to Logging.


Last update: 2020-11-27