Skip to content

Configuration Properties

The following contains the configuration properties of EventLoggingListener and HistoryServer.

spark.eventLog.dir

Directory where Spark events are logged to (e.g. hdfs://namenode:8021/directory)

Default: /tmp/spark-events

The directory must exist before SparkContext is created.

spark.eventLog.buffer.kb

Size of the buffer to use when writing to output streams.

Default: 100

spark.eventLog.compress

Whether to enable (true) or disable (false) event compression (using a CompressionCodec)

Default: false

spark.eventLog.enabled

Whether to enable (true) or disable (false) persisting Spark events.

Default: false

spark.eventLog.logBlockUpdates.enabled

Whether EventLoggingListener should log RDD block updates (true) or not (false)

Default: false

spark.eventLog.overwrite

Whether to enable (true) or disable (false) deleting (or at least overwriting) an existing .inprogress event log files

Default: false

spark.history.fs.logDirectory

The directory for event log files. The directory has to exist before starting History Server.

Default: file:/tmp/spark-events

spark.history.kerberos.enabled

Whether to enable (true) or disable (false) security when working with HDFS with security enabled (Kerberos).

Default: false

spark.history.kerberos.keytab

Keytab to use for login to Kerberos. Required when spark.history.kerberos.enabled is enabled.

Default: (empty)

spark.history.kerberos.principal

Kerberos principal. Required when spark.history.kerberos.enabled is enabled.

Default: (empty)

spark.history.provider

Fully-qualified class name of an ApplicationHistoryProvider for HistoryServer.

Default: org.apache.spark.deploy.history.FsHistoryProvider

spark.history.store.path

Local directory where to cache application history information (by )

Default: (undefined) (i.e. all history information will be kept in memory)

spark.history.retainedApplications

How many Spark applications HistoryServer should retain

Default: 50

spark.history.ui.maxApplications

How many Spark applications HistoryServer should show in the UI

Default: (unbounded)

spark.history.ui.port

The port of History Server's web UI.

Default: 18080


Last update: 2020-12-05
Back to top