Skip to content

== [[StorageStatus]] StorageStatus

StorageStatus is a developer API that Spark uses to pass "just enough" information about registered[BlockManagers] in a Spark application between Spark services (mostly for monitoring purposes like[web UI] or[]s).


There are two ways to access StorageStatus about all the known BlockManagers in a Spark application:


* Being a[] and intercepting[onBlockManagerAdded] and[onBlockManagerRemoved] events

StorageStatus is <> when:

  • BlockManagerMasterEndpoint[is requested for storage status] (of every[BlockManager] in a Spark application)

[[internal-registries]] .StorageStatus's Internal Registries and Counters [cols="1,2",options="header",width="100%"] |=== | Name | Description

| [[_nonRddBlocks]] _nonRddBlocks | Lookup table of BlockIds per BlockId.

Used when...FIXME

| [[_rddBlocks]] _rddBlocks | Lookup table of BlockIds with BlockStatus per RDD id.

Used when...FIXME |===

=== [[updateStorageInfo]] updateStorageInfo Internal Method

[source, scala]

updateStorageInfo( blockId: BlockId, newBlockStatus: BlockStatus): Unit


NOTE: updateStorageInfo is used when...FIXME

=== [[creating-instance]] Creating StorageStatus Instance

StorageStatus takes the following when created:

  • [[blockManagerId]][]
  • [[maxMem]] Maximum memory --[total available on-heap and off-heap memory for storage on the BlockManager]

StorageStatus initializes the <>.

=== [[rddBlocksById]] Getting RDD Blocks For RDD -- rddBlocksById Method

[source, scala]

rddBlocksById(rddId: Int): Map[BlockId, BlockStatus]

rddBlocksById gives the blocks (as BlockId with their status as BlockStatus) that belong to rddId RDD.

=== [[removeBlock]] Removing Block (From Internal Registries) -- removeBlock Internal Method

[source, scala]

removeBlock(blockId: BlockId): Option[BlockStatus]

removeBlock removes blockId from <<_rddBlocks, _rddBlocks>> registry and returns it.

Internally, removeBlock <> of blockId (to be empty, i.e. removed).

removeBlock branches off per the type of[], i.e. RDDBlockId or not.

For a RDDBlockId, removeBlock finds the RDD in <<_rddBlocks, _rddBlocks>> and removes the blockId. removeBlock removes the RDD (from <<_rddBlocks, _rddBlocks>>) completely, if there are no more blocks registered.

For a non-RDDBlockId, removeBlock removes blockId from <<_nonRddBlocks, _nonRddBlocks>> registry.

=== [[addBlock]] Registering Status of Data Block -- addBlock Method

[source, scala]

addBlock( blockId: BlockId, blockStatus: BlockStatus): Unit


NOTE: addBlock is used when...FIXME

=== [[getBlock]] getBlock Method

[source, scala]

getBlock(blockId: BlockId): Option[BlockStatus]


NOTE: getBlock is used when...FIXME