Skip to content


DiskStore manages data blocks on disk for BlockManager.

DiskStore and BlockManager

Creating Instance

DiskStore takes the following to be created:

DiskStore is created for BlockManager.

Block Sizes

blockSizes: ConcurrentHashMap[BlockId, Long]

DiskStore uses ConcurrentHashMap (Java) as a registry of blocks and the data size (on disk).

A new entry is added when put and moveFileToBlock.

An entry is removed when remove.


  blockId: BlockId,
  bytes: ChunkedByteBuffer): Unit

putBytes put the block and writes the buffer out (to the given channel).

putBytes is used when:


  blockId: BlockId): BlockData
  f: File,
  blockSize: Long): BlockData

getBytes requests the DiskBlockManager for the block file and the size.

getBytes requests the SecurityManager for getIOEncryptionKey and returns a EncryptedBlockData if available or a DiskBlockData otherwise.

getBytes is used when:


  blockId: BlockId): Long

getSize looks up the block in the blockSizes registry.

getSize is used when:


  sourceFile: File,
  blockSize: Long,
  targetBlockId: BlockId): Unit


moveFileToBlock is used when:

Checking if Block File Exists

  blockId: BlockId): Boolean

contains requests the DiskBlockManager for the block file and checks whether the file actually exists or not.

contains is used when:

Persisting Block to Disk

  blockId: BlockId)(
  writeFunc: WritableByteChannel => Unit): Unit

put prints out the following DEBUG message to the logs:

Attempting to put block [blockId]

put requests the DiskBlockManager for the block file for the input BlockId.

put opens the block file for writing (wrapped into a CountingWritableChannel to count the bytes written). put executes the given writeFunc function (with the WritableByteChannel of the block file) and saves the bytes written (to the blockSizes registry).

In the end, put prints out the following DEBUG message to the logs:

Block [fileName] stored as [size] file on disk in [time] ms

In case of any exception, put deletes the block file.

put throws an IllegalStateException when the block is already stored:

Block [blockId] is already present in the disk store

put is used when:

Removing Block

  blockId: BlockId): Boolean


remove is used when:

  • BlockManager is requested to removeBlockInternal
  • DiskStore is requested to put (and an IOException is thrown)


Enable ALL logging level for logger to see what happens inside.

Add the following line to conf/

Refer to Logging.

Back to top