HistoryServer — WebUI For Active And Completed Spark Applications

HistoryServer is an extension of the web UI for reviewing event logs of running (active) and completed Spark applications with event log collection enabled (based on spark.eventLog.enabled configuration property).

HistoryServer supports custom configuration properties.

HistoryServer is created when…​FIXME

HistoryServer uses the HttpServlet to handle requests to /* URI that FIXME.

HistoryServer is a ApplicationCacheOperations.

HistoryServer is a UIRoot.

HistoryServer uses spark.history.retainedApplications configuration property (default: 50) for…​FIXME

HistoryServer uses spark.history.ui.maxApplications configuration property (default: unbounded) for…​FIXME

Enable ALL logging level for org.apache.spark.deploy.history.HistoryServer logger to see what happens inside.

Add the following line to conf/log4j.properties:


Refer to Logging.

Creating HistoryServer Instance

HistoryServer takes the following to be created:

HistoryServer initializes the internal properties.

While being created, HistoryServer is requested to initialize.

Initializing HistoryServer — initialize Method

initialize(): Unit
initialize is part of WebUI Contract to initialize web components.


attachSparkUI Method

  appId: String,
  attemptId: Option[String],
  ui: SparkUI,
  completed: Boolean): Unit
attachSparkUI is part of ApplicationCacheOperations Contract to…​FIXME.


Launching HistoryServer Standalone Application — main Method

main(argStrings: Array[String]): Unit


Requesting Spark Application UI — getAppUI Method

getAppUI(appId: String, attemptId: Option[String]): Option[LoadedAppUI]
getAppUI is part of ApplicationCacheOperations Contract to…​FIXME.


withSparkUI Method

withSparkUI[T](appId: String, attemptId: Option[String])(fn: SparkUI => T): T
withSparkUI is part of UIRoot Contract to…​FIXME.


loadAppUi Internal Method

loadAppUi(appId: String, attemptId: Option[String]): Boolean


loadAppUi is used exclusively when HistoryServer is created.

doGet Method

doGet(req: HttpServletRequest, res: HttpServletResponse): Unit
doGet is part of Java Servlet’s HttpServlet to handle HTTP GET requests.


doGet is used when…​FIXME

Internal Properties

Name Description


ApplicationCache for this HistoryServer and retainedApplications

Used when…​FIXME


Java Servlets' HttpServlet

Used exclusively when HistoryServer is requested to initialize (and attaches the servlet to the web UI to handle /* URI)