BlockDataManager¶

BlockDataManager is an abstraction of block data managers that manage storage for blocks of data (aka block storage management API).

BlockDataManager uses BlockId to uniquely identify blocks of data and ManagedBuffer to represent them.

BlockDataManager is used to initialize a BlockTransferService.

BlockDataManager is used to create a NettyBlockRpcServer.

Contract¶

diagnoseShuffleBlockCorruption¶

diagnoseShuffleBlockCorruption(
  blockId: BlockId,
  checksumByReader: Long,
  algorithm: String): Cause

getHostLocalShuffleData¶

getHostLocalShuffleData(
  blockId: BlockId,
  dirs: Array[String]): ManagedBuffer

Used when:

• ShuffleBlockFetcherIterator is requested to fetchHostLocalBlock

getLocalBlockData¶

getLocalBlockData(
  blockId: BlockId): ManagedBuffer

Used when:

• NettyBlockRpcServer is requested to receive a request (OpenBlocks and FetchShuffleBlocks)

getLocalDiskDirs¶

getLocalDiskDirs: Array[String]

Used when:

• NettyBlockRpcServer is requested to handle a GetLocalDirsForExecutors request

putBlockData¶

putBlockData(
  blockId: BlockId,
  data: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Boolean

Stores (puts) a block data (as a ManagedBuffer) for the given BlockId. Returns true when completed successfully or false when failed.

Used when:

• NettyBlockRpcServer is requested to receive a UploadBlock request

putBlockDataAsStream¶

putBlockDataAsStream(
  blockId: BlockId,
  level: StorageLevel,
  classTag: ClassTag[_]): StreamCallbackWithID

Used when:

• NettyBlockRpcServer is requested to receiveStream

releaseLock¶

releaseLock(
  blockId: BlockId,
  taskContext: Option[TaskContext]): Unit

Used when:

• TorrentBroadcast is requested to releaseBlockManagerLock
• BlockManager is requested to handleLocalReadFailure, getLocalValues, getOrElseUpdate, doPut, releaseLockAndDispose

Implementations¶

• BlockManager