== [[StorageStatusListener]] StorageStatusListener -- Spark Listener for Tracking BlockManagers
StorageStatusListener
is a SparkListener.md[] that uses <
StorageStatusListener
is created and registered when SparkUI
spark-webui-SparkUI.md#create[is created]. It is later used to create spark-webui-executors-ExecutorsListener.md[ExecutorsListener] and spark-webui-StorageListener.md[StorageListener] Spark listeners.
[[SparkListener-callbacks]] .StorageStatusListener's SparkListener Callbacks (in alphabetical order) [cols="1,2",options="header",width="100%"] |=== | Callback | Description
| [[onBlockManagerAdded]] onBlockManagerAdded
| Adds an executor id with spark-blockmanager-StorageStatus.md[StorageStatus] (with storage:BlockManager.md[BlockManager] and maximum memory on the executor) to <
Removes any other BlockManager
that may have been registered for the executor earlier in <
| onBlockManagerRemoved
| Removes an executor from <
Removes the oldest spark-blockmanager-StorageStatus.md[StorageStatus] when the number of entries in <
| onBlockUpdated
| Updates spark-blockmanager-StorageStatus.md[StorageStatus] for an executor in <NONE
storage level] and updates otherwise.
| [[onUnpersistRDD]] onUnpersistRDD
| <BlockManager
registered as spark-blockmanager-StorageStatus.md[StorageStatus] in <
[[internal-registries]] .StorageStatusListener's Internal Registries and Counters [cols="1,2",options="header",width="100%"] |=== | Name | Description
| [[deadExecutorStorageStatus]] deadExecutorStorageStatus
| Collection of spark-blockmanager-StorageStatus.md[StorageStatus] of removed/inactive BlockManagers
.
Accessible using <
Adds an element when StorageStatusListener
<
Removes an element when StorageStatusListener
<
| [[executorIdToStorageStatus]] executorIdToStorageStatus
| Lookup table of spark-blockmanager-StorageStatus.md[StorageStatus] per executor (including the driver).
Adds an entry when StorageStatusListener
<
Removes an entry when StorageStatusListener
<
Updates StorageStatus
of an executor when StorageStatusListener
<
=== [[updateStorageStatus-executor]] Updating Storage Status For Executor -- updateStorageStatus
Method
CAUTION: FIXME
=== [[storageStatusList]] Active BlockManagers (on Executors) -- storageStatusList
Method
[source, scala]¶
storageStatusList: Seq[StorageStatus]¶
storageStatusList
gives a collection of spark-blockmanager-StorageStatus.md[StorageStatus] (from <
[NOTE]¶
storageStatusList
is used when:
StorageStatusListener
<> ExecutorsListener
does spark-webui-executors-ExecutorsListener.md#activeStorageStatusList[activeStorageStatusList]-
StorageListener
does spark-webui-StorageListener.md#activeStorageStatusList[activeStorageStatusList]¶
=== [[deadStorageStatusList]] deadStorageStatusList
Method
[source, scala]¶
deadStorageStatusList: Seq[StorageStatus]¶
deadStorageStatusList
gives <
NOTE: deadStorageStatusList
is used when ExecutorsListener
spark-webui-executors-ExecutorsListener.md#deadStorageStatusList[is requested for inactive/dead BlockManagers].
=== [[updateStorageStatus-unpersistedRDD]] Removing RDD Blocks for Unpersisted RDD -- updateStorageStatus
Internal Method
[source, scala]¶
updateStorageStatus(unpersistedRDDId: Int)¶
updateStorageStatus
takes <
updateStorageStatus
then spark-blockmanager-StorageStatus.md#rddBlocksById[finds RDD blocks] for unpersistedRDDId
RDD (for every BlockManager
) and spark-blockmanager-StorageStatus.md#removeBlock[removes the blocks].
NOTE: storageStatusList
is used exclusively when StorageStatusListener
<