Skip to content

BlockTransferService

BlockTransferService is an extension of the BlockStoreClient abstraction for shuffle clients that can fetch and upload blocks of data (synchronously or asynchronously).

BlockTransferService is a network service available by a host name and a port.

BlockTransferService was introduced in SPARK-3019 Pluggable block transfer interface (BlockTransferService).

Contract

Host Name

hostName: String

Host name this service is listening on

Used when:

Initializing

init(
  blockDataManager: BlockDataManager): Unit

Used when:

Port

port: Int

Used when:

Uploading Block Asynchronously

uploadBlock(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Future[Unit]

Used when:

Implementations

Uploading Block Synchronously

uploadBlockSync(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Unit

uploadBlockSync uploadBlock and waits till it finishes.

uploadBlockSync is used when:

  • BlockManager is requested to replicate
  • ShuffleMigrationRunnable is requested to run

Last update: 2021-07-23
Back to top