ExecutorAllocationManager can be used to dynamically allocate executors based on processing workload.
ExecutorAllocationManager intercepts Spark events using the internal ExecutorAllocationListener that keeps track of the workload.
ExecutorAllocationManager takes the following to be created:
- ContextCleaner (default:
validateSettings makes sure that the settings for dynamic allocation are correct.
validateSettings throws a
SparkException when the following are not met:
spark.dynamicAllocation.minExecutors must be positive
spark.dynamicAllocation.maxExecutors must be
spark.dynamicAllocation.executorIdleTimeout must be greater than
spark.shuffle.service.enabled must be enabled.
ExecutorAllocationManager uses ExecutorAllocationManagerSource for performance metrics.
ExecutorMonitor is used for the performance metrics:
- numberExecutorsPendingToRemove (based on pendingRemovalCount)
- numberAllExecutors (based on executorCount)
ExecutorMonitor is used for the following:
- timedOutExecutors when
ExecutorAllocationManageris requested to schedule
- executorCount when
ExecutorAllocationManageris requested to addExecutors
- executorCount, pendingRemovalCount and executorsKilled when
ExecutorAllocationManageris requested to removeExecutors
ExecutorAllocationListener is used to calculate the maximum number of executors needed.
Maximum Number of Executors Needed¶
maxNumExecutorsNeeded is the smallest integer value that is greater than or equal to the multiplication of the total number of pending and running tasks by executorAllocationRatio divided by tasksPerExecutorForFullParallelism.
maxNumExecutorsNeeded is used for:
start requests the ScheduledExecutorService to schedule the
The schedule delay of
100 is not configurable.
start is used when
SparkContext is created.
If there are executors to be removed,
schedule turns the initializing internal flag off.
schedule updateAndSyncNumExecutorsTarget with the current time.
In the end,
schedule removes the executors to be removed if there are any.
updateAndSyncNumExecutorsTarget( now: Long): Int
stop shuts down <
stop waits 10 seconds for the termination to be complete.
stop is used when
SparkContext is requested to stop
spark-dynamic-executor-allocation Allocation Executor¶
spark-dynamic-executor-allocation allocation executor is a...FIXME
removeExecutors( executors: Seq[(String, Int)]): Seq[String]
removeExecutors is used when:
ExecutorAllocationManageris requested to schedule executors
ALL logging level for
org.apache.spark.ExecutorAllocationManager logger to see what happens inside.
Add the following line to
Refer to Logging.