Spark History Server

Spark History Server is the web UI for completed and running (aka incomplete) Spark applications. It is an extension of Spark’s web UI.

spark history server webui
Figure 1. History Server’s web UI
Enable collecting events in your Spark applications using spark.eventLog.enabled Spark property.

You can start History Server by executing shell script and stop it using accepts --properties-file [propertiesFile] command-line option that specifies the properties file with the custom Spark properties.

$ ./sbin/ --properties-file

If not specified explicitly, Spark History Server uses the default configuration file, i.e. spark-defaults.conf.

Enable INFO logging level for org.apache.spark.deploy.history logger to see what happens inside.

Add the following line to conf/

Refer to Logging.

Starting History Server — script

You can start a HistoryServer instance by executing $SPARK_HOME/sbin/ script (where SPARK_HOME is the directory of your Spark installation).

$ ./sbin/
starting org.apache.spark.deploy.history.HistoryServer, logging to .../spark/logs/spark-jacek-org.apache.spark.deploy.history.HistoryServer-1-japila.out

Internally, script starts org.apache.spark.deploy.history.HistoryServer standalone application for execution (using shell script).

$ ./bin/spark-class org.apache.spark.deploy.history.HistoryServer
Using the more explicit approach with spark-class to start Spark History Server could be easier to trace execution by seeing the logs printed out to the standard output and hence terminal directly.

When started, it prints out the following INFO message to the logs:

INFO HistoryServer: Started daemon with process name: [processName]

It registers signal handlers (using SignalUtils) for TERM, HUP, INT to log their execution:

ERROR HistoryServer: RECEIVED SIGNAL [signal]

It inits security if enabled (using spark.history.kerberos.enabled setting).

FIXME Describe initSecurity

It creates a SecurityManager.

It creates a HistoryServer and requests it to bind to spark.history.ui.port port.

The host’s IP can be specified using SPARK_LOCAL_IP environment variable (defaults to

You should see the following INFO message in the logs:

INFO HistoryServer: Bound HistoryServer to [host], and started at [webUrl]

It registers a shutdown hook to call stop on the HistoryServer instance.

Use shell script to to stop a running History Server.

Stopping History Server — script

You can stop a running instance of HistoryServer using $SPARK_HOME/sbin/ shell script.

$ ./sbin/
stopping org.apache.spark.deploy.history.HistoryServer


Table 1. Spark Properties
Setting Default Value Description



The port of the History Server’s UI.



The directory with the event logs. The directory has to exist before starting History Server.



How many Spark applications to retain.



how many Spark applications to show in the UI.



Enable security when working with HDFS with security enabled (Kerberos).



Kerberos principal. Required when spark.history.kerberos.enabled is enabled.



Keytab to use for login to Kerberos. Required when spark.history.kerberos.enabled is enabled.



The fully-qualified class name for a ApplicationHistoryProvider.