Skip to content

TransportConf

Creating Instance

TransportConf takes the following to be created:

TransportConf is created when:

  • SparkTransportConf utility is used to fromSparkConf
  • YarnShuffleService (Spark on YARN) is requested to serviceInit

Module Name

TransportConf is given the name of a module the transport-related configuration properties are for and is as follows (per SparkTransportConf):

getModuleName

String getModuleName()

getModuleName returns the module name.

getConfKey

String getConfKey(
  String suffix)

getConfKey creates the key of a configuration property (with the module and the given suffix):

spark.[module].[suffix]

Suffixes

io.mode

  • nio (default)
  • epoll

io.preferDirectBufs

Controls whether Spark prefers allocating off-heap byte buffers within Netty (true) or not (false).

Default: true

io.connectionTimeout

io.connectionCreationTimeout

io.backLog

The requested maximum length of the queue of incoming connections

Default: -1 (no backlog)

io.numConnectionsPerPeer

Default: 1

io.serverThreads

io.clientThreads

Default: 0

io.receiveBuffer

io.sendBuffer

sasl.timeout

io.maxRetries

io.retryWait

Time that we will wait in order to perform a retry after an IOException. Only relevant if maxIORetries is greater than 0.

Default: 5s

io.lazyFD

io.enableVerboseMetrics

Enables Netty's memory detailed metrics

Default: false

io.enableTcpKeepAlive

preferDirectBufsForSharedByteBufAllocators

The value of spark.network.io.preferDirectBufs.

sharedByteBufAllocators

The value of spark.network.sharedByteBufAllocators.enabled.

Back to top