EventLoggingListener¶

EventLoggingListener is a SparkListener that writes out JSON-encoded events of a Spark application with event logging enabled (based on spark.eventLog.enabled configuration property).

EventLoggingListener supports custom configuration properties.

EventLoggingListener writes out log files to a directory (based on spark.eventLog.dir configuration property).

Creating Instance¶

EventLoggingListener takes the following to be created:

• Application ID
• Application Attempt ID
• Log Directory
• SparkConf
• Hadoop Configuration

EventLoggingListener is created when SparkContext is created (with spark.eventLog.enabled enabled).

EventLogFileWriter¶

logWriter: EventLogFileWriter

EventLoggingListener creates a EventLogFileWriter when created.

The EventLogFileWriter is started when EventLoggingListener is started.

The EventLogFileWriter is stopped when EventLoggingListener is stopped.

The EventLogFileWriter is requested to writeEvent when EventLoggingListener is requested to start and log an event.

Starting EventLoggingListener¶

start(): Unit

start requests the EventLogFileWriter to start and initEventLog.

initEventLog¶

initEventLog(): Unit

initEventLog...FIXME

Logging Event¶

logEvent(
  event: SparkListenerEvent,
  flushLogger: Boolean = false): Unit

logEvent persists the given SparkListenerEvent in JSON format.

logEvent converts the event to JSON format and requests the EventLogFileWriter to write it out.

Stopping EventLoggingListener¶

stop(): Unit

stop requests the EventLogFileWriter to stop.

stop is used when SparkContext is requested to stop.

inprogress File Extension¶

EventLoggingListener uses .inprogress file extension for in-flight event log files of active Spark applications.

Logging¶

Enable ALL logging level for org.apache.spark.scheduler.EventLoggingListener logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.scheduler.EventLoggingListener=ALL

Refer to Logging.