Skip to content

SparkTransportConf Utility

fromSparkConf

fromSparkConf(
  _conf: SparkConf,
  module: String, // (1)
  numUsableCores: Int = 0,
  role: Option[String] = None): TransportConf // (2)
  1. The given module is shuffle most of the time except:
  2. Only defined in NettyRpcEnv to be either driver or executor

fromSparkConf makes a copy (clones) the given SparkConf.

fromSparkConf sets the following configuration properties (for the given module):

  • spark.[module].io.serverThreads
  • spark.[module].io.clientThreads

The values are taken using the following properties in the order and until one is found (with suffix being serverThreads or clientThreads, respectively):

  1. spark.[role].[module].io.[suffix]
  2. spark.[module].io.[suffix]

Unless found, fromSparkConf defaults to the default number of threads (based on the given numUsableCores and not more than 8).

In the end, fromSparkConf creates a TransportConf (for the given module and the updated SparkConf).

fromSparkConf is used when:

Back to top