SparkTransportConf Utility¶

fromSparkConf¶

fromSparkConf(
  _conf: SparkConf,
  module: String, // (1)
  numUsableCores: Int = 0,
  role: Option[String] = None): TransportConf // (2)

1. The given module is shuffle most of the time except:
   • rpc for NettyRpcEnv
   • files for NettyRpcEnv
2. Only defined in NettyRpcEnv to be either driver or executor

fromSparkConf makes a copy (clones) the given SparkConf.

fromSparkConf sets the following configuration properties (for the given module):

• spark.[module].io.serverThreads
• spark.[module].io.clientThreads

The values are taken using the following properties in the order and until one is found (with suffix being serverThreads or clientThreads, respectively):

1. spark.[role].[module].io.[suffix]
2. spark.[module].io.[suffix]

Unless found, fromSparkConf defaults to the default number of threads (based on the given numUsableCores and not more than 8).

In the end, fromSparkConf creates a TransportConf (for the given module and the updated SparkConf).

fromSparkConf is used when:

• SparkEnv utility is used to create a SparkEnv (with the spark.shuffle.service.enabled configuration property enabled)
• ExternalShuffleService is created
• NettyBlockTransferService is requested to init
• NettyRpcEnv is created and requested for a downloadClient
• IndexShuffleBlockResolver is created
• ShuffleBlockPusher is requested to initiateBlockPush
• BlockManager is requested to readDiskBlockFromSameHostExecutor