Configuration Properties¶

The following contains the configuration properties of EventLoggingListener and HistoryServer.

spark.eventLog¶

buffer.kb¶

spark.eventLog.buffer.kb

Size of the buffer to use when writing to output streams

Default: 100k

compress¶

spark.eventLog.compress

Enables event compression (using a CompressionCodec)

Default: false

compression.codec¶

spark.eventLog.compression.codec

The codec used to compress event log (with spark.eventLog.compress enabled). By default, Spark provides four codecs: lz4, lzf, snappy, and zstd. You can also use fully qualified class names to specify the codec.

Default: zstd

dir¶

spark.eventLog.dir

Directory where Spark events are logged to (e.g. hdfs://namenode:8021/directory)

Default: /tmp/spark-events

The directory must exist before SparkContext can be created

enabled¶

spark.eventLog.enabled

Enables persisting Spark events

Default: false

erasureCoding.enabled¶

spark.eventLog.erasureCoding.enabled

Default: false

gcMetrics.youngGenerationGarbageCollectors¶

spark.eventLog.gcMetrics.youngGenerationGarbageCollectors

Names of supported young generation garbage collectors. A name usually is the output of GarbageCollectorMXBean.getName.

Default: Copy, PS Scavenge, ParNew, G1 Young Generation (the built-in young generation garbage collectors)

gcMetrics.oldGenerationGarbageCollectors¶

spark.eventLog.gcMetrics.oldGenerationGarbageCollectors

Names of supported old generation garbage collectors. A name usually is the output of GarbageCollectorMXBean.getName.

Default: MarkSweepCompact, PS MarkSweep, ConcurrentMarkSweep, G1 Old Generation (the built-in old generation garbage collectors)

logBlockUpdates.enabled¶

spark.eventLog.logBlockUpdates.enabled

Enables log RDD block updates using EventLoggingListener

Default: false

logStageExecutorMetrics¶

spark.eventLog.logStageExecutorMetrics

Enables logging of per-stage peaks of executor metrics (for each executor) to the event log

Default: false

longForm.enabled¶

spark.eventLog.longForm.enabled

Default: false

overwrite¶

spark.eventLog.overwrite

Enables deleting (or at least overwriting) an existing .inprogress event log files

Default: false

rolling.enabled¶

spark.eventLog.rolling.enabled

Enables rolling over event log files. When enabled, cuts down each event log file to spark.eventLog.rolling.maxFileSize

Default: false

rolling.maxFileSize¶

spark.eventLog.rolling.maxFileSize

Max size of event log file to be rolled over (with spark.eventLog.rolling.enabled enabled)

Default: 128m

Must be at least 10 MiB

spark.history¶

fs.logDirectory¶

spark.history.fs.logDirectory

The directory for event log files. The directory has to exist before starting History Server.

Default: file:/tmp/spark-events

kerberos.enabled¶

spark.history.kerberos.enabled

Whether to enable (true) or disable (false) security when working with HDFS with security enabled (Kerberos).

Default: false

kerberos.keytab¶

spark.history.kerberos.keytab

Keytab to use for login to Kerberos. Required when spark.history.kerberos.enabled is enabled.

Default: (empty)

kerberos.principal¶

spark.history.kerberos.principal

Kerberos principal. Required when spark.history.kerberos.enabled is enabled.

Default: (empty)

provider¶

spark.history.provider

Fully-qualified class name of an ApplicationHistoryProvider for HistoryServer.

Default: org.apache.spark.deploy.history.FsHistoryProvider

retainedApplications¶

spark.history.retainedApplications

How many Spark applications HistoryServer should retain

Default: 50

store.path¶

spark.history.store.path

Local directory where to cache application history information (by )

Default: (undefined) (i.e. all history information will be kept in memory)

ui.maxApplications¶

spark.history.ui.maxApplications

How many Spark applications HistoryServer should show in the UI

Default: (unbounded)

ui.port¶

spark.history.ui.port

The port of History Server's web UI.

Default: 18080