BlockTransferService

BlockTransferService is an extension of the ShuffleClient abstraction for shuffle clients that can fetch and upload blocks of data synchronously or asynchronously.

BlockTransferService is a networking service available by a host name and a port.

Contract

close

close(): Unit

Used when BlockManager is requested to stop

fetchBlocks

fetchBlocks(
  host: String,
  port: Int,
  execId: String,
  blockIds: Array[String],
  listener: BlockFetchingListener,
  tempFileManager: DownloadFileManager): Unit

Fetches a sequence of blocks from a remote node asynchronously

fetchBlocks is part of the ShuffleClient abstraction.

Used when BlockTransferService is requested to fetch only one block (in a blocking fashion)

hostName

hostName: String

Used when BlockManager is requested to initialize

init

init(
  blockDataManager: BlockDataManager): Unit

Used when BlockManager is requested to initialize (with the BlockDataManager being the BlockManager itself)

port

port: Int

Used when BlockManager is requested to initialize

uploadBlock

uploadBlock(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Future[Unit]

Used when BlockTransferService is requested to upload a single block to a remote node (in a blocking fashion)

BlockTransferServices

NettyBlockTransferService is the default and only known BlockTransferService.

fetchBlockSync Method

fetchBlockSync(
  host: String,
  port: Int,
  execId: String,
  blockId: String,
  tempFileManager: TempFileManager): ManagedBuffer

fetchBlockSync…​FIXME

Synchronous (and hence blocking) fetchBlockSync to fetch one block blockId (that corresponds to the ShuffleClient parent’s asynchronous fetchBlocks).

fetchBlockSync is a mere wrapper around fetchBlocks to fetch one blockId block that waits until the fetch finishes.

fetchBlockSync is used when…​FIXME

Uploading Single Block to Remote Node (Blocking Fashion)

uploadBlockSync(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Unit

uploadBlockSync…​FIXME

uploadBlockSync is a mere blocking wrapper around uploadBlock that waits until the upload finishes.

uploadBlockSync is used when BlockManager is requested to replicate (when a replication level is greater than 1).