TransportConf¶

Creating Instance¶

TransportConf takes the following to be created:

• Module Name
• ConfigProvider

TransportConf is created when:

• SparkTransportConf utility is used to fromSparkConf
• YarnShuffleService (Spark on YARN) is requested to serviceInit

Module Name¶

TransportConf is given the name of a module the transport-related configuration properties are for and is as follows (per SparkTransportConf):

• shuffle
• rpc for NettyRpcEnv
• files for NettyRpcEnv

getModuleName¶

String getModuleName()

getModuleName returns the module name.

getConfKey¶

String getConfKey(
  String suffix)

getConfKey creates the key of a configuration property (with the module and the given suffix):

spark.[module].[suffix]

Suffixes¶

io.mode¶

• nio (default)
• epoll

io.preferDirectBufs¶

Controls whether Spark prefers allocating off-heap byte buffers within Netty (true) or not (false).

Default: true

io.connectionTimeout¶

io.connectionCreationTimeout¶

io.backLog¶

The requested maximum length of the queue of incoming connections

Default: -1 (no backlog)

io.numConnectionsPerPeer¶

Default: 1

io.serverThreads¶

io.clientThreads¶

Default: 0

io.receiveBuffer¶

io.sendBuffer¶

sasl.timeout¶

io.maxRetries¶

io.retryWait¶

Time that we will wait in order to perform a retry after an IOException. Only relevant if maxIORetries is greater than 0.

Default: 5s

io.lazyFD¶

io.enableVerboseMetrics¶

Enables Netty's memory detailed metrics

Default: false

io.enableTcpKeepAlive¶

preferDirectBufsForSharedByteBufAllocators¶

The value of spark.network.io.preferDirectBufs.

sharedByteBufAllocators¶

The value of spark.network.sharedByteBufAllocators.enabled.