Skip to content

ExecutorAllocationClient

ExecutorAllocationClient is a <> for clients to communicate with a cluster manager to request or kill executors.

=== [[contract]] ExecutorAllocationClient Contract

[source, scala]

trait ExecutorAllocationClient { def getExecutorIds(): Seq[String] def requestTotalExecutors(numExecutors: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]): Boolean def requestExecutors(numAdditionalExecutors: Int): Boolean def killExecutor(executorId: String): Boolean def killExecutors(executorIds: Seq[String]): Seq[String] def killExecutorsOnHost(host: String): Boolean }


NOTE: ExecutorAllocationClient is a private[spark] contract.

.ExecutorAllocationClient Contract [cols="1,2",options="header",width="100%"] |=== | Method | Description

| [[getExecutorIds]] getExecutorIds | Finds identifiers of the executors in use.

Used when ROOT:SparkContext.md#getExecutorIds[SparkContext calculates the executors in use] and also when spark-streaming/spark-streaming-dynamic-allocation.md[Spark Streaming manages executors].

| [[requestTotalExecutors]] requestTotalExecutors a| Updates the cluster manager with the exact number of executors desired. It returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

  • ROOT:SparkContext.md#requestTotalExecutors[SparkContext requests executors] (for coarse-grained scheduler backends only).

  • ExecutorAllocationManager spark-ExecutorAllocationManager.md#start[starts], does spark-ExecutorAllocationManager.md#updateAndSyncNumExecutorsTarget[updateAndSyncNumExecutorsTarget], and spark-ExecutorAllocationManager.md#addExecutors[addExecutors].

  • Streaming ExecutorAllocationManager spark-streaming/spark-streaming-ExecutorAllocationManager.md#requestExecutors[requests executors].

  • yarn/spark-yarn-yarnschedulerbackend.md#stop[YarnSchedulerBackend stops].

| [[requestExecutors]] requestExecutors | Requests additional executors from a cluster manager and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when ROOT:SparkContext.md#requestExecutors[SparkContext requests additional executors] (for coarse-grained scheduler backends only).

| [[killExecutor]] killExecutor a| Requests a cluster manager to kill a single executor that is no longer in use and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

The default implementation simply calls <> (with a single-element collection of executors to kill).

Used when:

  • ExecutorAllocationManager spark-ExecutorAllocationManager.md#removeExecutor[removes an executor].

  • SparkContext ROOT:SparkContext.md#killExecutors[is requested to kill executors].

  • Streaming ExecutorAllocationManager spark-streaming/spark-streaming-ExecutorAllocationManager.md#killExecutor[is requested to kill executors].

| [[killExecutors]] killExecutors | Requests that a cluster manager to kill one or many executors that are no longer in use and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Interestingly, it is only used for <>.

| [[killExecutorsOnHost]] killExecutorsOnHost | Used exclusively when BlacklistTracker kills blacklisted executors.

|===


Last update: 2020-10-06