Skip to content

ExternalShuffleBlockHandler

= ExternalShuffleBlockHandler

ExternalShuffleBlockHandler is a network:RpcHandler.md[].

When created, ExternalShuffleBlockHandler requires a network:OneForOneStreamManager.md[] and network:TransportConf.md[] with a registeredExecutorFile to create a ExternalShuffleBlockResolver.

It <BlockTransferMessage messages>>: <> and <>.

== [[handleMessage]] Handling Messages

[source, java]

handleMessage( BlockTransferMessage msgObj, TransportClient client, RpcResponseCallback callback)


handleMessage handles two types of BlockTransferMessage messages:

  • <>
  • <>

For any other BlockTransferMessage message it throws a UnsupportedOperationException:

Unexpected message: [msgObj]

== [[OpenBlocks]] OpenBlocks

[source, java]

OpenBlocks( String appId, String execId, String[] blockIds)


When OpenBlocks is received, <> authorizes the client.

CAUTION: FIXME checkAuth?

It then <> for each block id in blockIds (using <>).

Finally, it network:OneForOneStreamManager.md#registerStream[registers a stream] and does callback.onSuccess with a serialized byte buffer (for the streamId and the number of blocks in msg).

CAUTION: FIXME callback.onSuccess?

You should see the following TRACE message in the logs:

Registered streamId [streamId] with [length] buffers for client [clientId] from host [remoteAddress]

== [[RegisterExecutor]] RegisterExecutor

[source, java]

RegisterExecutor( String appId, String execId, ExecutorShuffleInfo executorInfo)


RegisterExecutor...FIXME

== [[logging]] Logging

Enable ALL logging level for org.apache.spark.network.shuffle.ExternalShuffleBlockHandler logger to see what happens inside.

Add the following line to conf/log4j.properties:

[source,plaintext]

log4j.logger.org.apache.spark.network.shuffle.ExternalShuffleBlockHandler=ALL

Refer to ROOT:spark-logging.md[Logging].


Last update: 2020-10-06