ExecutorAllocationClient

ExecutorAllocationClient is a contract for clients to communicate with a cluster manager to request or kill executors.

ExecutorAllocationClient Contract

trait ExecutorAllocationClient {
  def getExecutorIds(): Seq[String]
  def requestTotalExecutors(numExecutors: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]): Boolean
  def requestExecutors(numAdditionalExecutors: Int): Boolean
  def killExecutor(executorId: String): Boolean
  def killExecutors(executorIds: Seq[String]): Seq[String]
  def killExecutorsOnHost(host: String): Boolean
}
ExecutorAllocationClient is a private[spark] contract.
Table 1. ExecutorAllocationClient Contract
Method Description

getExecutorIds

Finds identifiers of the executors in use.

Used when SparkContext calculates the executors in use and also when Spark Streaming manages executors.

requestTotalExecutors

Updates the cluster manager with the exact number of executors desired. It returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

requestExecutors

Requests additional executors from a cluster manager and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when SparkContext requests additional executors (for coarse-grained scheduler backends only).

killExecutor

Requests a cluster manager to kill a single executor that is no longer in use and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

The default implementation simply calls killExecutors (with a single-element collection of executors to kill).

Used when:

killExecutors

Requests that a cluster manager to kill one or many executors that are no longer in use and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Interestingly, it is only used for killExecutor.

killExecutorsOnHost

Used exclusively when BlacklistTracker kills blacklisted executors.