RpcUtils¶

Maximum Message Size¶

maxMessageSizeBytes(
  conf: SparkConf): Int

maxMessageSizeBytes is the value of spark.rpc.message.maxSize configuration property in bytes (by multiplying the value by 1024 * 1024).

maxMessageSizeBytes throws an IllegalArgumentException when the value is above 2047 MB:

spark.rpc.message.maxSize should not be greater than 2047 MB

maxMessageSizeBytes is used when:

• MapOutputTrackerMaster is requested for the maxRpcMessageSize
• Executor is requested for the maxDirectResultSize
• CoarseGrainedSchedulerBackend is requested for the maxRpcMessageSize

makeDriverRef¶

makeDriverRef(
  name: String,
  conf: SparkConf,
  rpcEnv: RpcEnv): RpcEndpointRef

makeDriverRef...FIXME

makeDriverRef is used when:

• BarrierTaskContext is created
• SparkEnv utility is used to create a SparkEnv (on executors)
• Executor is created
• PluginContextImpl is requested for driverEndpoint