Skip to content


SparkUI is a WebUI of Spark applications.

Welcome Page of web UI — Jobs Tab

Creating Instance

SparkUI takes the following to be created:

While being created, SparkUI initializes itself.

SparkUI is created using create utility.

UI Port

  conf: SparkConf): Int

getUIPort requests the SparkConf for the value of spark.ui.port configuration property.

getUIPort is used when:

Creating SparkUI

  sc: Option[SparkContext],
  store: AppStatusStore,
  conf: SparkConf,
  securityManager: SecurityManager,
  appName: String,
  basePath: String,
  startTime: Long,
  appSparkVersion: String): SparkUI

create creates a new SparkUI with appSparkVersion being the current Spark version.

create is used when:


initialize(): Unit

initialize is part of the WebUI abstraction.

initialize creates and attaches the following tabs:

  1. JobsTab
  2. StagesTab
  3. StorageTab
  4. EnvironmentTab
  5. ExecutorsTab

initialize attaches itself as the UIRoot.

initialize attaches the PrometheusResource for executor metrics based on spark.ui.prometheus.enabled configuration property.


SparkUI is an UIRoot

Review Me

SparkUI is <> along with the following:

  • SparkContext is created (for a live Spark application with[spark.ui.enabled] configuration property enabled)

  • FsHistoryProvider is requested for the[application UI] (for a live or completed Spark application)

.Creating SparkUI for Live Spark Application image::spark-webui-SparkUI.png[align="center"]

When <> (while SparkContext is created for a live Spark application), SparkUI gets the following:

  • Live AppStatusStore (with a ElementTrackingStore using an[] and a AppStatusListener for a live Spark application)

  • Name of the Spark application that is exactly the value of[] configuration property

  • Empty base path

When started, SparkUI binds to <> address that you can control using SPARK_PUBLIC_DNS environment variable or[] Spark property.

NOTE: With[spark.ui.killEnabled] configuration property turned on, SparkUI <> (subject to SecurityManager.checkModifyPermissions permissions).

SparkUI gets an <> that is then used for the following:

  • <>, i.e.[JobsTab],[StagesTab],[StorageTab],[EnvironmentTab]

  • AbstractApplicationResource is requested for[jobsList],[oneJob],[executorList],[allExecutorList],[rddList],[rddData],[environmentInfo]

  • StagesResource is requested for[stageList],[stageData],[oneAttemptData],[taskSummary],[taskList]

  • SparkUI is requested for the current <>

  • Creating Spark SQL's SQLTab (when SQLHistoryServerPlugin is requested to setupUI)

  • Spark Streaming's BatchPage is created

[[internal-registries]] .SparkUI's Internal Properties (e.g. Registries, Counters and Flags) [cols="1,2",options="header",width="100%"] |=== | Name | Description

| appId | [[appId]] |===


Enable INFO logging level for org.apache.spark.ui.SparkUI logger to see what happens inside.

Add the following line to conf/

Refer to[Logging].

== [[setAppId]] Assigning Unique Identifier of Spark Application -- setAppId Method

[source, scala]

setAppId(id: String): Unit

setAppId sets the internal <>.

setAppId is used when SparkContext is created.

== [[stop]] Stopping SparkUI -- stop Method

[source, scala]

stop(): Unit

stop stops the HTTP server and prints the following INFO message to the logs:

INFO SparkUI: Stopped Spark web UI at [appUIAddress]

NOTE: appUIAddress in the above INFO message is the result of <> method.

== [[appUIAddress]] appUIAddress Method

[source, scala]

appUIAddress: String

appUIAddress returns the entire URL of a Spark application's web UI, including http:// scheme.

Internally, appUIAddress uses <>.

== [[createLiveUI]] createLiveUI Method

[source, scala]

createLiveUI( sc: SparkContext, conf: SparkConf, listenerBus: SparkListenerBus, jobProgressListener: JobProgressListener, securityManager: SecurityManager, appName: String, startTime: Long): SparkUI

createLiveUI creates a SparkUI for a live running Spark application.

Internally, createLiveUI simply forwards the call to <>.

createLiveUI is used when SparkContext is created.

== [[createHistoryUI]] createHistoryUI Method


== [[appUIHostPort]] appUIHostPort Method

[source, scala]

appUIHostPort: String

appUIHostPort returns the Spark application's web UI which is the public hostname and port, excluding the scheme.

NOTE: <> uses appUIHostPort and adds http:// scheme.

== [[getAppName]] getAppName Method

[source, scala]

getAppName: String

getAppName returns the name of the Spark application (of a SparkUI instance).

NOTE: getAppName is used when...FIXME

== [[create]] Creating SparkUI Instance -- create Factory Method

[source, scala]

create( sc: Option[SparkContext], store: AppStatusStore, conf: SparkConf, securityManager: SecurityManager, appName: String, basePath: String = "", startTime: Long, appSparkVersion: String = org.apache.spark.SPARK_VERSION): SparkUI

create creates a SparkUI backed by a[].

Internally, create simply creates a new <> (with the predefined Spark version).

create is used when:

  • SparkContext is created
  • FsHistoryProvider is requested to[getAppUI] (for a Spark application that already finished)

Creating Instance

SparkUI takes the following when created:

  • [[store]][]
  • [[sc]][]
  • [[conf]][SparkConf]
  • [[securityManager]] SecurityManager
  • [[appName]] Application name
  • [[basePath]] basePath
  • [[startTime]] Start time
  • [[appSparkVersion]] appSparkVersion

SparkUI initializes the <> and <>.

== [[initialize]] Attaching Tabs and Context Handlers -- initialize Method

[source, scala]

initialize(): Unit

NOTE: initialize is part of[WebUI Contract] to initialize web components.

initialize creates and <> the following tabs (with the reference to the SparkUI and its <>):

.[StagesTab] .[StorageTab] .[EnvironmentTab] .[ExecutorsTab]

In the end, initialize creates and[attaches] the following ServletContextHandlers:

.[Creates a static handler] for serving files from a static directory, i.e. /static to serve static files from org/apache/spark/ui/static directory (on CLASSPATH)

.[Creates the /api/* context handler] for the[Status REST API]

.[Creates a redirect handler] to redirect /jobs/job/kill to /jobs/ and request the JobsTab to execute handleKillRequest before redirection

.[Creates a redirect handler] to redirect /stages/stage/kill to /stages/ and request the StagesTab to execute[handleKillRequest] before redirection