ExecutorAllocationManager can be used to dynamically allocate executors based on processing workload.
ExecutorAllocationManager intercepts Spark events using the internal ExecutorAllocationListener that keeps track of the workload.
ExecutorAllocationManager takes the following to be created:
- ContextCleaner (default:
ExecutorAllocationManager is created when SparkContext is created.
validateSettings makes sure that the settings for dynamic allocation are correct.
validateSettings throws a
SparkException when the following are not met:
spark.dynamicAllocation.minExecutors must be positive
spark.dynamicAllocation.maxExecutors must be
spark.dynamicAllocation.executorIdleTimeout must be greater than
spark.shuffle.service.enabled must be enabled.
ExecutorAllocationManager uses ExecutorAllocationManagerSource for performance metrics.
ExecutorMonitor is used for the performance metrics:
- numberExecutorsPendingToRemove (based on pendingRemovalCount)
- numberAllExecutors (based on executorCount)
ExecutorMonitor is used for the following:
- timedOutExecutors when
ExecutorAllocationManageris requested to schedule
- executorCount when
ExecutorAllocationManageris requested to addExecutors
- executorCount, pendingRemovalCount and executorsKilled when
ExecutorAllocationManageris requested to removeExecutors
ExecutorAllocationListener is used to calculate the maximum number of executors needed.
Maximum Number of Executors Needed¶
maxNumExecutorsNeeded is the smallest integer value that is greater than or equal to the multiplication of the total number of pending and running tasks by executorAllocationRatio divided by tasksPerExecutorForFullParallelism.
maxNumExecutorsNeeded is used for:
start requests the ScheduledExecutorService to schedule the
The schedule delay of
100 is not configurable.
start is used when
SparkContext is created.
schedule calls <
It then go over <
updateAndSyncNumExecutorsTarget( now: Long): Int
stop shuts down <
stop waits 10 seconds for the termination to be complete.
stop is used when
SparkContext is requested to stop
spark-dynamic-executor-allocation Allocation Executor¶
spark-dynamic-executor-allocation allocation executor is a...FIXME
ALL logging level for
org.apache.spark.ExecutorAllocationManager logger to see what happens inside.
Add the following line to
Refer to Logging.