ExecutorAllocationClient¶

ExecutorAllocationClient is an abstraction of schedulers that can communicate with a cluster manager to request or kill executors.

Contract¶

Active Executor IDs¶

getExecutorIds(): Seq[String]

Used when:

• SparkContext is requested for active executors

isExecutorActive¶

isExecutorActive(
  id: String): Boolean

Whether a given executor (by ID) is active (and can be used to execute tasks)

Used when:

• FIXME

Killing Executors¶

killExecutors(
  executorIds: Seq[String],
  adjustTargetNumExecutors: Boolean,
  countFailures: Boolean,
  force: Boolean = false): Seq[String]

Requests a cluster manager to kill given executors and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

• ExecutorAllocationClient is requested to kill an executor
• ExecutorAllocationManager is requested to removeExecutors
• SparkContext is requested to kill executors and killAndReplaceExecutor
• BlacklistTracker is requested to kill an executor
• DriverEndpoint is requested to handle a KillExecutorsOnHost message

Killing Executors on Host¶

killExecutorsOnHost(
  host: String): Boolean

Used when:

• BlacklistTracker is requested to kill executors on a blacklisted node

Requesting Additional Executors¶

requestExecutors(
  numAdditionalExecutors: Int): Boolean

Requests additional executors from a cluster manager and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

• SparkContext is requested for additional executors

Updating Total Executors¶

requestTotalExecutors(
  resourceProfileIdToNumExecutors: Map[Int, Int],
  numLocalityAwareTasksPerResourceProfileId: Map[Int, Int],
  hostToLocalTaskCount: Map[Int, Map[String, Int]]): Boolean

Updates a cluster manager with the exact number of executors desired. Returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

• SparkContext is requested to update the number of total executors

• ExecutorAllocationManager is requested to start, updateAndSyncNumExecutorsTarget, addExecutors, removeExecutors

Implementations¶

• CoarseGrainedSchedulerBackend
• KubernetesClusterSchedulerBackend (Spark on Kubernetes)
• MesosCoarseGrainedSchedulerBackend
• StandaloneSchedulerBackend ([Spark Standalone]https://books.japila.pl/spark-standalone-internals/StandaloneSchedulerBackend))
• YarnSchedulerBackend

Killing Single Executor¶

killExecutor(
  executorId: String): Boolean

killExecutor kill the given executor.

killExecutor is used when:

• ExecutorAllocationManager removes an executor.
• SparkContext is requested to kill executors.

Decommissioning Executors¶

decommissionExecutors(
  executorsAndDecomInfo: Array[(String, ExecutorDecommissionInfo)],
  adjustTargetNumExecutors: Boolean,
  triggeredByExecutor: Boolean): Seq[String]

decommissionExecutors kills the given executors.

decommissionExecutors is used when:

• ExecutorAllocationClient is requested to decommission a single executor
• ExecutorAllocationManager is requested to remove executors
• StandaloneSchedulerBackend (Spark Standalone) is requested to executorDecommissioned

Decommissioning Single Executor¶

decommissionExecutor(
  executorId: String,
  decommissionInfo: ExecutorDecommissionInfo,
  adjustTargetNumExecutors: Boolean,
  triggeredByExecutor: Boolean = false): Boolean

decommissionExecutor...FIXME

decommissionExecutor is used when:

• DriverEndpoint is requested to handle a ExecutorDecommissioning message