Configuration Properties

The following contains the configuration properties of EventLoggingListener and Spark History Server.

EventLoggingListener is responsible for writing out JSON-encoded events of a Spark application to an event log file that HistoryServer can display in a web UI-based interface.
Table 1. EventLoggingListener’s Spark Properties
Name Description

spark.eventLog.buffer.kb

Size of the buffer to use when writing to output streams.

Default: 100

spark.eventLog.compress

Whether to enable (true) or disable (false) event compression (using a CompressionCodec)

Default: false

spark.eventLog.dir

Directory where Spark events are logged to (e.g. hdfs://namenode:8021/directory)

Default: /tmp/spark-events

The directory must exist before Spark starts up.

spark.eventLog.enabled

Whether to enable (true) or disable (false) persisting Spark events.

Default: false

spark.eventLog.logBlockUpdates.enabled

Whether EventLoggingListener should log RDD block updates (true) or not (false)

Default: false

spark.eventLog.overwrite

Whether to enable (true) or disable (false) deleting (or at least overwriting) an existing .inprogress event log files

Default: false

spark.eventLog.testing

(internal) Whether to enable (true) or disable (false) adding JSON-encoded events to the internal loggedEvents array for testing

Default: false

Table 2. HistoryServer’s Spark Properties
Name Description

spark.history.fs.logDirectory

The directory for event log files. The directory has to exist before starting History Server.

Default: file:/tmp/spark-events

spark.history.kerberos.enabled

Whether to enable (true) or disable (false) security when working with HDFS with security enabled (Kerberos).

Default: false

spark.history.kerberos.keytab

Keytab to use for login to Kerberos. Required when spark.history.kerberos.enabled is enabled.

Default: (empty)

spark.history.kerberos.principal

Kerberos principal. Required when spark.history.kerberos.enabled is enabled.

Default: (empty)

spark.history.provider

Fully-qualified class name of the ApplicationHistoryProvider

Default: org.apache.spark.deploy.history.FsHistoryProvider

spark.history.retainedApplications

How many Spark applications HistoryServer should retain

Default: 50

spark.history.ui.maxApplications

How many Spark applications HistoryServer should show in the UI

Default: (unbounded)

spark.history.ui.port

The port of History Server’s web UI.

Default: 18080