HistoryServer — WebUI For Active And Completed Spark Applications

HistoryServer is an extension of the web UI for reviewing event logs of running (active) and completed Spark applications with event log collection enabled (based on spark.eventLog.enabled configuration property).

HistoryServer supports custom configuration properties.

HistoryServer is created when…​FIXME

HistoryServer uses the HttpServlet to handle requests to /* URI that FIXME.

HistoryServer is a ApplicationCacheOperations.

HistoryServer is a UIRoot.

HistoryServer uses spark.history.retainedApplications configuration property (default: 50) for…​FIXME

HistoryServer uses spark.history.ui.maxApplications configuration property (default: unbounded) for…​FIXME

Enable ALL logging level for org.apache.spark.deploy.history.HistoryServer logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.deploy.history.HistoryServer=ALL

Refer to Logging.

Creating HistoryServer Instance

HistoryServer takes the following to be created:

HistoryServer initializes the internal properties.

While being created, HistoryServer is requested to initialize.

Initializing HistoryServer — initialize Method

initialize(): Unit
initialize is part of WebUI Contract to initialize web components.

initialize…​FIXME

attachSparkUI Method

attachSparkUI(
  appId: String,
  attemptId: Option[String],
  ui: SparkUI,
  completed: Boolean): Unit
attachSparkUI is part of ApplicationCacheOperations Contract to…​FIXME.

attachSparkUI…​FIXME

Launching HistoryServer Standalone Application — main Method

main(argStrings: Array[String]): Unit

main…​FIXME

Requesting Spark Application UI — getAppUI Method

getAppUI(appId: String, attemptId: Option[String]): Option[LoadedAppUI]
getAppUI is part of ApplicationCacheOperations Contract to…​FIXME.

getAppUI…​FIXME

withSparkUI Method

withSparkUI[T](appId: String, attemptId: Option[String])(fn: SparkUI => T): T
withSparkUI is part of UIRoot Contract to…​FIXME.

withSparkUI…​FIXME

loadAppUi Internal Method

loadAppUi(appId: String, attemptId: Option[String]): Boolean

loadAppUi…​FIXME

loadAppUi is used exclusively when HistoryServer is created.

doGet Method

doGet(req: HttpServletRequest, res: HttpServletResponse): Unit
doGet is part of Java Servlet’s HttpServlet to handle HTTP GET requests.

doGet…​FIXME

doGet is used when…​FIXME

Internal Properties

Name Description

appCache

ApplicationCache for this HistoryServer and retainedApplications

Used when…​FIXME

loaderServlet

Java Servlets' HttpServlet

Used exclusively when HistoryServer is requested to initialize (and attaches the servlet to the web UI to handle /* URI)