RpcEnvConfig

RpcEnvConfig is a configuration of an RpcEnv:

RpcEnvConfig is created when RpcEnv utility is used to create an RpcEnv (using RpcEnvFactory).

Client Mode

When an RPC Environment is initialized as part of the initialization of the driver or executors (using RpcEnv.create), clientMode is false for the driver and true for executors.

"Client mode" means the RPC env will not listen for incoming connections.

This allows certain processes in the Spark stack (such as Executors or tha YARN client-mode AM) to act as pure clients when using the netty-based RPC backend, reducing the number of sockets Spark apps need to use and also the number of open ports.

The AM connects to the driver in "client mode", and that connection is used for all driver — AM communication, and so the AM is properly notified when the connection goes down.

In "general", non-YARN case, clientMode flag is therefore enabled for executors and disabled for the driver.

In Spark on YARN in client deploy mode, clientMode flag is however enabled explicitly when Spark on YARN’s ApplicationMaster creates the sparkYarnAM RPC Environment.