ReplayListenerBus

ReplayListenerBus is used by FsHistoryProvider.

Replaying JSON-encoded SparkListenerEvents from Stream

replay(
  logData: InputStream,
  sourceName: String,
  maybeTruncated: Boolean = false): Unit

replay reads JSON-encoded SparkListenerEvent events from logData (one event per line) and posts them to all registered SparkListenerInterface listeners.

replay uses jackson from json4s library to parse the AST for JSON.

When there is an exception parsing a JSON event, you may see the following WARN message in the logs (for the last line) or a JsonParseException.

WARN Got JsonParseException from log file $sourceName at line [lineNumber], the file might not have finished writing cleanly.

Any other non-IO exceptions end up with the following ERROR messages in the logs:

ERROR Exception parsing Spark event log: [sourceName]
ERROR Malformed line #[lineNumber]: [currentLine]
The sourceName input argument is only used for messages.