Skip to content

SparkUI

SparkUI is a WebUI of Spark applications.

Welcome Page of web UI — Jobs Tab

Creating Instance

SparkUI takes the following to be created:

While being created, SparkUI initializes itself.

SparkUI is created using create utility.

UI Port

getUIPort(
  conf: SparkConf): Int

getUIPort requests the SparkConf for the value of spark.ui.port configuration property.

getUIPort is used when:

Creating SparkUI

create(
  sc: Option[SparkContext],
  store: AppStatusStore,
  conf: SparkConf,
  securityManager: SecurityManager,
  appName: String,
  basePath: String,
  startTime: Long,
  appSparkVersion: String): SparkUI

create creates a new SparkUI with appSparkVersion being the current Spark version.

create is used when:

Initializing

initialize(): Unit

initialize is part of the WebUI abstraction.

initialize creates and attaches the following tabs:

  1. JobsTab
  2. StagesTab
  3. StorageTab
  4. EnvironmentTab
  5. ExecutorsTab

initialize attaches itself as the UIRoot.

initialize attaches the PrometheusResource for executor metrics based on spark.ui.prometheus.enabled configuration property.

UIRoot

SparkUI is an UIRoot

Review Me

SparkUI is <> along with the following:

  • SparkContext is created (for a live Spark application with spark-webui-properties.md#spark.ui.enabled[spark.ui.enabled] configuration property enabled)

  • FsHistoryProvider is requested for the spark-history-server:FsHistoryProvider.md#getAppUI[application UI] (for a live or completed Spark application)

.Creating SparkUI for Live Spark Application image::spark-webui-SparkUI.png[align="center"]

When <> (while SparkContext is created for a live Spark application), SparkUI gets the following:

  • Live AppStatusStore (with a ElementTrackingStore using an core:InMemoryStore.md[] and a AppStatusListener for a live Spark application)

  • Name of the Spark application that is exactly the value of SparkConf.md#spark.app.name[spark.app.name] configuration property

  • Empty base path

When started, SparkUI binds to <> address that you can control using SPARK_PUBLIC_DNS environment variable or spark-driver.md#spark_driver_host[spark.driver.host] Spark property.

NOTE: With spark-webui-properties.md#spark.ui.killEnabled[spark.ui.killEnabled] configuration property turned on, SparkUI <> (subject to SecurityManager.checkModifyPermissions permissions).

SparkUI gets an <> that is then used for the following:

  • <>, i.e. JobsTab.md#creating-instance[JobsTab], spark-webui-StagesTab.md#creating-instance[StagesTab], spark-webui-StorageTab.md#creating-instance[StorageTab], spark-webui-EnvironmentTab.md#creating-instance[EnvironmentTab]

  • AbstractApplicationResource is requested for spark-api-AbstractApplicationResource.md#jobsList[jobsList], spark-api-AbstractApplicationResource.md#oneJob[oneJob], spark-api-AbstractApplicationResource.md#executorList[executorList], spark-api-AbstractApplicationResource.md#allExecutorList[allExecutorList], spark-api-AbstractApplicationResource.md#rddList[rddList], spark-api-AbstractApplicationResource.md#rddData[rddData], spark-api-AbstractApplicationResource.md#environmentInfo[environmentInfo]

  • StagesResource is requested for spark-api-StagesResource.md#stageList[stageList], spark-api-StagesResource.md#stageData[stageData], spark-api-StagesResource.md#oneAttemptData[oneAttemptData], spark-api-StagesResource.md#taskSummary[taskSummary], spark-api-StagesResource.md#taskList[taskList]

  • SparkUI is requested for the current <>

  • Creating Spark SQL's SQLTab (when SQLHistoryServerPlugin is requested to setupUI)

  • Spark Streaming's BatchPage is created

[[internal-registries]] .SparkUI's Internal Properties (e.g. Registries, Counters and Flags) [cols="1,2",options="header",width="100%"] |=== | Name | Description

| appId | [[appId]] |===

[TIP]

Enable INFO logging level for org.apache.spark.ui.SparkUI logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.ui.SparkUI=INFO

Refer to spark-logging.md[Logging].

== [[setAppId]] Assigning Unique Identifier of Spark Application -- setAppId Method

[source, scala]

setAppId(id: String): Unit

setAppId sets the internal <>.

setAppId is used when SparkContext is created.

== [[stop]] Stopping SparkUI -- stop Method

[source, scala]

stop(): Unit

stop stops the HTTP server and prints the following INFO message to the logs:

INFO SparkUI: Stopped Spark web UI at [appUIAddress]

NOTE: appUIAddress in the above INFO message is the result of <> method.

== [[appUIAddress]] appUIAddress Method

[source, scala]

appUIAddress: String

appUIAddress returns the entire URL of a Spark application's web UI, including http:// scheme.

Internally, appUIAddress uses <>.

== [[createLiveUI]] createLiveUI Method

[source, scala]

createLiveUI( sc: SparkContext, conf: SparkConf, listenerBus: SparkListenerBus, jobProgressListener: JobProgressListener, securityManager: SecurityManager, appName: String, startTime: Long): SparkUI

createLiveUI creates a SparkUI for a live running Spark application.

Internally, createLiveUI simply forwards the call to <>.

createLiveUI is used when SparkContext is created.

== [[createHistoryUI]] createHistoryUI Method

CAUTION: FIXME

== [[appUIHostPort]] appUIHostPort Method

[source, scala]

appUIHostPort: String

appUIHostPort returns the Spark application's web UI which is the public hostname and port, excluding the scheme.

NOTE: <> uses appUIHostPort and adds http:// scheme.

== [[getAppName]] getAppName Method

[source, scala]

getAppName: String

getAppName returns the name of the Spark application (of a SparkUI instance).

NOTE: getAppName is used when...FIXME

== [[create]] Creating SparkUI Instance -- create Factory Method

[source, scala]

create( sc: Option[SparkContext], store: AppStatusStore, conf: SparkConf, securityManager: SecurityManager, appName: String, basePath: String = "", startTime: Long, appSparkVersion: String = org.apache.spark.SPARK_VERSION): SparkUI

create creates a SparkUI backed by a core:AppStatusStore.md[].

Internally, create simply creates a new <> (with the predefined Spark version).

create is used when:

  • SparkContext is created
  • FsHistoryProvider is requested to spark-history-server:FsHistoryProvider.md#getAppUI[getAppUI] (for a Spark application that already finished)

Creating Instance

SparkUI takes the following when created:

  • [[store]] core:AppStatusStore.md[]
  • [[sc]] SparkContext.md[]
  • [[conf]] SparkConf.md[SparkConf]
  • [[securityManager]] SecurityManager
  • [[appName]] Application name
  • [[basePath]] basePath
  • [[startTime]] Start time
  • [[appSparkVersion]] appSparkVersion

SparkUI initializes the <> and <>.

== [[initialize]] Attaching Tabs and Context Handlers -- initialize Method

[source, scala]

initialize(): Unit

NOTE: initialize is part of spark-webui-WebUI.md#initialize[WebUI Contract] to initialize web components.

initialize creates and <> the following tabs (with the reference to the SparkUI and its <>):

. spark-webui-StagesTab.md[StagesTab] . spark-webui-StorageTab.md[StorageTab] . spark-webui-EnvironmentTab.md[EnvironmentTab] . spark-webui-ExecutorsTab.md[ExecutorsTab]

In the end, initialize creates and spark-webui-WebUI.md#attachHandler[attaches] the following ServletContextHandlers:

. spark-webui-JettyUtils.md#createStaticHandler[Creates a static handler] for serving files from a static directory, i.e. /static to serve static files from org/apache/spark/ui/static directory (on CLASSPATH)

. spark-api-ApiRootResource.md#getServletHandler[Creates the /api/* context handler] for the spark-api.md[Status REST API]

. spark-webui-JettyUtils.md#createRedirectHandler[Creates a redirect handler] to redirect /jobs/job/kill to /jobs/ and request the JobsTab to execute handleKillRequest before redirection

. spark-webui-JettyUtils.md#createRedirectHandler[Creates a redirect handler] to redirect /stages/stage/kill to /stages/ and request the StagesTab to execute spark-webui-StagesTab.md#handleKillRequest[handleKillRequest] before redirection