Skip to content

FsHistoryProvider

FsHistoryProvider is the default ApplicationHistoryProvider for Spark History Server.

Creating Instance

FsHistoryProvider takes the following to be created:

FsHistoryProvider is created when HistoryServer standalone application is started (and no spark.history.provider configuration property was defined).

Path of Application History Cache

storePath: Option[File]

FsHistoryProvider uses spark.history.store.path configuration property for the directory to cache application history.

With storePath defined, FsHistoryProvider uses a LevelDB as the KVStore. Otherwise, a InMemoryStore.

With storePath defined, FsHistoryProvider uses a HistoryServerDiskManager as the disk manager.

Disk Manager

diskManager: Option[HistoryServerDiskManager]

FsHistoryProvider creates a HistoryServerDiskManager when created (with storePath defined based on spark.history.store.path configuration property).

FsHistoryProvider uses the HistoryServerDiskManager for the following:

SparkUI of Spark Application

getAppUI(
  appId: String,
  attemptId: Option[String]): Option[LoadedAppUI]

getAppUI is part of the ApplicationHistoryProvider abstraction.

getAppUI...FIXME

onUIDetached

onUIDetached(): Unit

onUIDetached is part of the ApplicationHistoryProvider abstraction.

onUIDetached...FIXME

loadDiskStore

loadDiskStore(
  dm: HistoryServerDiskManager,
  appId: String,
  attempt: AttemptInfoWrapper): KVStore

loadDiskStore...FIXME

loadDiskStore is used in getAppUI (with HistoryServerDiskManager available).

createInMemoryStore

createInMemoryStore(
  attempt: AttemptInfoWrapper): KVStore

createInMemoryStore...FIXME

createInMemoryStore is used in getAppUI.

rebuildAppStore

rebuildAppStore(
  store: KVStore,
  reader: EventLogFileReader,
  lastUpdated: Long): Unit

rebuildAppStore...FIXME

rebuildAppStore is used in loadDiskStore and createInMemoryStore.

cleanAppData

cleanAppData(
  appId: String,
  attemptId: Option[String],
  logPath: String): Unit

cleanAppData...FIXME

cleanAppData is used in checkForLogs and deleteAttemptLogs.

Polling for Logs

startPolling(): Unit

startPolling...FIXME

startPolling is used in initialize and startSafeModeCheckThread.

Checking Available Event Logs

checkForLogs(): Unit

checkForLogs...FIXME

Logging

Enable ALL logging level for org.apache.spark.deploy.history.FsHistoryProvider logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.deploy.history.FsHistoryProvider=ALL

Refer to Logging.