Skip to content

SchedulerBackendUtils Utility

Default Number of Executors

SchedulerBackendUtils defaults to 2 as the default number of executors.

getInitialTargetExecutorNumber

getInitialTargetExecutorNumber(
  conf: SparkConf,
  numExecutors: Int = DEFAULT_NUMBER_EXECUTORS): Int

getInitialTargetExecutorNumber branches off based on whether Dynamic Allocation of Executors is enabled or not.

With no Dynamic Allocation of Executors, getInitialTargetExecutorNumber uses the spark.executor.instances configuration property (if defined) or uses the given numExecutors (and the DEFAULT_NUMBER_EXECUTORS).

With Dynamic Allocation of Executors enabled, getInitialTargetExecutorNumber getDynamicAllocationInitialExecutors and makes sure that the value is between the following configuration properties:

getInitialTargetExecutorNumber is used when:

  • KubernetesClusterSchedulerBackend (Spark on Kubernetes) is created
  • Spark on YARN's YarnAllocator, YarnClientSchedulerBackend and YarnClusterSchedulerBackend are used
Back to top