Skip to content

== [[StorageStatusListener]] StorageStatusListener -- Spark Listener for Tracking BlockManagers

StorageStatusListener is a[] that uses <> to track status of every[BlockManager] in a Spark application.

StorageStatusListener is created and registered when SparkUI[is created]. It is later used to create[ExecutorsListener] and[StorageListener] Spark listeners.

[[SparkListener-callbacks]] .StorageStatusListener's SparkListener Callbacks (in alphabetical order) [cols="1,2",options="header",width="100%"] |=== | Callback | Description

| [[onBlockManagerAdded]] onBlockManagerAdded | Adds an executor id with[StorageStatus] (with[BlockManager] and maximum memory on the executor) to <> internal registry.

Removes any other BlockManager that may have been registered for the executor earlier in <> internal registry.

| onBlockManagerRemoved | Removes an executor from <> internal registry and adds the removed[StorageStatus] to <> internal registry.

Removes the oldest[StorageStatus] when the number of entries in <> is bigger than[spark.ui.retainedDeadExecutors].

| onBlockUpdated | Updates[StorageStatus] for an executor in <> internal registry, i.e. removes a block for[NONE storage level] and updates otherwise.

| [[onUnpersistRDD]] onUnpersistRDD | <> for an unpersisted RDD (on every BlockManager registered as[StorageStatus] in <> internal registry). |===

[[internal-registries]] .StorageStatusListener's Internal Registries and Counters [cols="1,2",options="header",width="100%"] |=== | Name | Description

| [[deadExecutorStorageStatus]] deadExecutorStorageStatus | Collection of[StorageStatus] of removed/inactive BlockManagers.

Accessible using <> method.

Adds an element when StorageStatusListener <> (possibly removing one element from the head when the number of elements are above[spark.ui.retainedDeadExecutors] property).

Removes an element when StorageStatusListener <> (per executor) so the executor is not longer dead.

| [[executorIdToStorageStatus]] executorIdToStorageStatus | Lookup table of[StorageStatus] per executor (including the driver).

Adds an entry when StorageStatusListener <>.

Removes an entry when StorageStatusListener <>.

Updates StorageStatus of an executor when StorageStatusListener <>. |===

=== [[updateStorageStatus-executor]] Updating Storage Status For Executor -- updateStorageStatus Method


=== [[storageStatusList]] Active BlockManagers (on Executors) -- storageStatusList Method

[source, scala]

storageStatusList: Seq[StorageStatus]

storageStatusList gives a collection of[StorageStatus] (from <> internal registry).


storageStatusList is used when:

  • StorageStatusListener <>
  • ExecutorsListener does[activeStorageStatusList]
  • StorageListener does[activeStorageStatusList]

=== [[deadStorageStatusList]] deadStorageStatusList Method

[source, scala]

deadStorageStatusList: Seq[StorageStatus]

deadStorageStatusList gives <> internal registry.

NOTE: deadStorageStatusList is used when ExecutorsListener[is requested for inactive/dead BlockManagers].

=== [[updateStorageStatus-unpersistedRDD]] Removing RDD Blocks for Unpersisted RDD -- updateStorageStatus Internal Method

[source, scala]

updateStorageStatus(unpersistedRDDId: Int)

updateStorageStatus takes <>.

updateStorageStatus then[finds RDD blocks] for unpersistedRDDId RDD (for every BlockManager) and[removes the blocks].

NOTE: storageStatusList is used exclusively when StorageStatusListener <>.

Last update: 2020-11-27