OneForOneBlockFetcher¶

Creating Instance¶

OneForOneBlockFetcher takes the following to be created:

• TransportClient
• Application ID
• Executor ID
• Block IDs (String[])
• BlockFetchingListener
• TransportConf
• DownloadFileManager

OneForOneBlockFetcher is created when:

• NettyBlockTransferService is requested to fetch blocks
• ExternalBlockStoreClient is requested to fetch blocks

createFetchShuffleBlocksMsg¶

FetchShuffleBlocks createFetchShuffleBlocksMsg(
  String appId,
  String execId,
  String[] blockIds)

createFetchShuffleBlocksMsg...FIXME

Starting¶

void start()

start requests the TransportClient to sendRpc the BlockTransferMessage

start...FIXME

start is used when:

• ExternalBlockStoreClient is requested to fetchBlocks
• NettyBlockTransferService is requested to fetchBlocks

Logging¶

Enable ALL logging level for org.apache.spark.network.shuffle.OneForOneBlockFetcher logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.network.shuffle.OneForOneBlockFetcher=ALL

Refer to Logging.