ExecutorAllocationClient¶
ExecutorAllocationClient
is an abstraction of schedulers that can communicate with a cluster manager to request or kill executors.
Implementations¶
- CoarseGrainedSchedulerBackend
KubernetesClusterSchedulerBackend
(Spark on Kubernetes)- MesosCoarseGrainedSchedulerBackend
- StandaloneSchedulerBackend
- YarnSchedulerBackend
Active Executors¶
getExecutorIds(): Seq[String]
Used when:
SparkContext
is requested for active executors
Whether Executor is Active¶
isExecutorActive(
id: String): Boolean
Returns whether a given executor is active (and can be used to execute tasks)
Killing Executors¶
killExecutors(
executorIds: Seq[String],
adjustTargetNumExecutors: Boolean,
countFailures: Boolean,
force: Boolean = false): Seq[String]
Requests a cluster manager to kill given executors and returns whether the request has been acknowledged by the cluster manager (true
) or not (false
).
Used when:
ExecutorAllocationClient
is requested to kill an executorExecutorAllocationManager
is requested to removeExecutorsSparkContext
is requested to kill executors and killAndReplaceExecutorBlacklistTracker
is requested to kill an executorDriverEndpoint
is requested to handle a KillExecutorsOnHost message
Killing Executors on Host¶
killExecutorsOnHost(
host: String): Boolean
Used when:
BlacklistTracker
is requested to kill executors on a blacklisted node
Requesting Additional Executors¶
requestExecutors(
numAdditionalExecutors: Int): Boolean
Requests additional executors from a cluster manager and returns whether the request has been acknowledged by the cluster manager (true
) or not (false
).
Used when:
SparkContext
is requested for additional executors
Updating Total Executors¶
requestTotalExecutors(
numExecutors: Int,
localityAwareTasks: Int,
hostToLocalTaskCount: Map[String, Int]): Boolean
Updates a cluster manager with the exact number of executors desired. Returns whether the request has been acknowledged by the cluster manager (true
) or not (false
).
Used when:
-
SparkContext
is requested to update the number of total executors -
ExecutorAllocationManager
is requested to start, updateAndSyncNumExecutorsTarget, addExecutors, removeExecutors
Killing Executor¶
killExecutor(
executorId: String): Boolean
killExecutor
kill the given executor.
killExecutor
is used when:
ExecutorAllocationManager
removes an executor.SparkContext
is requested to kill executors.