SchedulerBackendUtils Utility¶

Default Number of Executors¶

SchedulerBackendUtils defaults to 2 as the default number of executors.

getInitialTargetExecutorNumber¶

getInitialTargetExecutorNumber(
  conf: SparkConf,
  numExecutors: Int = DEFAULT_NUMBER_EXECUTORS): Int

getInitialTargetExecutorNumber branches off based on whether Dynamic Allocation of Executors is enabled or not.

With no Dynamic Allocation of Executors, getInitialTargetExecutorNumber uses the spark.executor.instances configuration property (if defined) or uses the given numExecutors (and the DEFAULT_NUMBER_EXECUTORS).

With Dynamic Allocation of Executors enabled, getInitialTargetExecutorNumber getDynamicAllocationInitialExecutors and makes sure that the value is between the following configuration properties:

• spark.dynamicAllocation.minExecutors
• spark.dynamicAllocation.maxExecutors

getInitialTargetExecutorNumber is used when:

• KubernetesClusterSchedulerBackend (Spark on Kubernetes) is created
• Spark on YARN's YarnAllocator, YarnClientSchedulerBackend and YarnClusterSchedulerBackend are used