BlockDataManager

BlockDataManager is an abstraction of block data managers that manage storage for blocks of data (aka block storage management API).

BlockDataManager uses BlockId to uniquely identify blocks of data and ManagedBuffer to represent them.

BlockDataManager is used to initialize a storage:BlockTransferService.adoc#init.

BlockDataManager is used to create a NettyBlockRpcServer.

Contract

getBlockData

getBlockData(
  blockId: BlockId): ManagedBuffer

Fetches a block data (as a ManagedBuffer) for the given BlockId

Used when:

putBlockData

putBlockData(
  blockId: BlockId,
  data: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Boolean

Stores (puts) a block data (as a ManagedBuffer) for the given BlockId. Returns true when completed successfully or false when failed.

Used when NettyBlockRpcServer is requested to handle an UploadBlock message

putBlockDataAsStream

putBlockDataAsStream(
  blockId: BlockId,
  level: StorageLevel,
  classTag: ClassTag[_]): StreamCallbackWithID

Stores a block data that will be received as a stream

Used when NettyBlockRpcServer is requested to receiveStream

releaseLock

releaseLock(
  blockId: BlockId,
  taskAttemptId: Option[Long]): Unit

Releases a lock

Used when:

Available BlockDataManagers

BlockManager is the default and only known BlockDataManager in Apache Spark.