Skip to content


SparkUI is the web UI of a Spark application (aka Application UI).

SparkUI is <> along with the following:

  • SparkContext is created (for a live Spark application with[spark.ui.enabled] configuration property enabled)

  • FsHistoryProvider is requested for the[application UI] (for a live or completed Spark application)

.Creating SparkUI for Live Spark Application image::spark-webui-SparkUI.png[align="center"]

When <> (while SparkContext is created for a live Spark application), SparkUI gets the following:

  • Live AppStatusStore (with a[] using an[] and a AppStatusListener for a live Spark application)

  • Name of the Spark application that is exactly the value of[] configuration property

  • Empty base path

When started, SparkUI binds to <> address that you can control using SPARK_PUBLIC_DNS environment variable or[] Spark property.

NOTE: With[spark.ui.killEnabled] configuration property turned on, SparkUI <> (subject to SecurityManager.checkModifyPermissions permissions).

SparkUI gets an <> that is then used for the following:

  • <>, i.e.[JobsTab],[StagesTab],[StorageTab],[EnvironmentTab]

  • AbstractApplicationResource is requested for[jobsList],[oneJob],[executorList],[allExecutorList],[rddList],[rddData],[environmentInfo]

  • StagesResource is requested for[stageList],[stageData],[oneAttemptData],[taskSummary],[taskList]

  • SparkUI is requested for the current <>

  • Creating Spark SQL's SQLTab (when SQLHistoryServerPlugin is requested to setupUI)

  • Spark Streaming's BatchPage is created

[[internal-registries]] .SparkUI's Internal Properties (e.g. Registries, Counters and Flags) [cols="1,2",options="header",width="100%"] |=== | Name | Description

| appId | [[appId]] |===


Enable INFO logging level for org.apache.spark.ui.SparkUI logger to see what happens inside.

Add the following line to conf/

Refer to[Logging].

== [[setAppId]] Assigning Unique Identifier of Spark Application -- setAppId Method

[source, scala]

setAppId(id: String): Unit

setAppId sets the internal <>.

setAppId is used when SparkContext is created.

== [[stop]] Stopping SparkUI -- stop Method

[source, scala]

stop(): Unit

stop stops the HTTP server and prints the following INFO message to the logs:

INFO SparkUI: Stopped Spark web UI at [appUIAddress]

NOTE: appUIAddress in the above INFO message is the result of <> method.

== [[appUIAddress]] appUIAddress Method

[source, scala]

appUIAddress: String

appUIAddress returns the entire URL of a Spark application's web UI, including http:// scheme.

Internally, appUIAddress uses <>.

== [[getSparkUser]] Accessing Spark User -- getSparkUser Method

[source, scala]

getSparkUser: String

getSparkUser returns the name of the user a Spark application runs as.

Internally, getSparkUser requests System property from[EnvironmentListener] Spark listener.

NOTE: getSparkUser is used...FIXME

== [[createLiveUI]] createLiveUI Method

[source, scala]

createLiveUI( sc: SparkContext, conf: SparkConf, listenerBus: SparkListenerBus, jobProgressListener: JobProgressListener, securityManager: SecurityManager, appName: String, startTime: Long): SparkUI

createLiveUI creates a SparkUI for a live running Spark application.

Internally, createLiveUI simply forwards the call to <>.

createLiveUI is used when SparkContext is created.

== [[createHistoryUI]] createHistoryUI Method


== [[appUIHostPort]] appUIHostPort Method

[source, scala]

appUIHostPort: String

appUIHostPort returns the Spark application's web UI which is the public hostname and port, excluding the scheme.

NOTE: <> uses appUIHostPort and adds http:// scheme.

== [[getAppName]] getAppName Method

[source, scala]

getAppName: String

getAppName returns the name of the Spark application (of a SparkUI instance).

NOTE: getAppName is used when...FIXME

== [[create]] Creating SparkUI Instance -- create Factory Method

[source, scala]

create( sc: Option[SparkContext], store: AppStatusStore, conf: SparkConf, securityManager: SecurityManager, appName: String, basePath: String = "", startTime: Long, appSparkVersion: String = org.apache.spark.SPARK_VERSION): SparkUI

create creates a SparkUI backed by a[].

Internally, create simply creates a new <> (with the predefined Spark version).

create is used when:

  • SparkContext is created
  • FsHistoryProvider is requested to[getAppUI] (for a Spark application that already finished)

Creating Instance

SparkUI takes the following when created:

  • [[store]][]
  • [[sc]][]
  • [[conf]][SparkConf]
  • [[securityManager]] SecurityManager
  • [[appName]] Application name
  • [[basePath]] basePath
  • [[startTime]] Start time
  • [[appSparkVersion]] appSparkVersion

SparkUI initializes the <> and <>.

== [[initialize]] Attaching Tabs and Context Handlers -- initialize Method

[source, scala]

initialize(): Unit

NOTE: initialize is part of[WebUI Contract] to initialize web components.

initialize creates and <> the following tabs (with the reference to the SparkUI and its <>):

.[JobsTab] .[StagesTab] .[StorageTab] .[EnvironmentTab] .[ExecutorsTab]

In the end, initialize creates and[attaches] the following ServletContextHandlers:

.[Creates a static handler] for serving files from a static directory, i.e. /static to serve static files from org/apache/spark/ui/static directory (on CLASSPATH)

.[Creates a redirect handler] to redirect / to /jobs/ (and so the[Jobs tab] is the welcome tab when you open the web UI)

.[Creates the /api/* context handler] for the[Status REST API]

.[Creates a redirect handler] to redirect /jobs/job/kill to /jobs/ and request the JobsTab to execute[handleKillRequest] before redirection

.[Creates a redirect handler] to redirect /stages/stage/kill to /stages/ and request the StagesTab to execute[handleKillRequest] before redirection

Last update: 2020-11-29