Spark Configuration Properties¶

spark.dynamicAllocation¶

cachedExecutorIdleTimeout¶

spark.dynamicAllocation.cachedExecutorIdleTimeout

How long (in seconds) to keep blocks cached

Default: The largest value representable as an Int

Must be >= 0

Used when:

• ExecutorMonitor is created
• RDD is requested to localCheckpoint (simply to print out a WARN message)

enabled¶

spark.dynamicAllocation.enabled

Enables Dynamic Allocation of Executors

Default: false

Used when:

• BarrierJobAllocationFailed is requested for ERROR_MESSAGE_RUN_BARRIER_WITH_DYN_ALLOCATION (for reporting purposes)
• RDD is requested to localCheckpoint (for reporting purposes)
• SparkSubmitArguments is requested to loadEnvironmentArguments (for validation purposes)
• Utils is requested to isDynamicAllocationEnabled

executorAllocationRatio¶

spark.dynamicAllocation.executorAllocationRatio

Default: 1.0

Must be between 0 (exclusive) and 1.0 (inclusive)

Used when:

• ExecutorAllocationManager is created

executorIdleTimeout¶

spark.dynamicAllocation.executorIdleTimeout

Default: 60

initialExecutors¶

spark.dynamicAllocation.initialExecutors

Default: spark.dynamicAllocation.minExecutors

maxExecutors¶

spark.dynamicAllocation.maxExecutors

Default: Int.MaxValue

minExecutors¶

spark.dynamicAllocation.minExecutors

Default: 0

schedulerBacklogTimeout¶

spark.dynamicAllocation.schedulerBacklogTimeout

(in seconds)

Default: 1

shuffleTracking.enabled¶

spark.dynamicAllocation.shuffleTracking.enabled

Default: false

Used when:

• ExecutorMonitor is created

shuffleTracking.timeout¶

spark.dynamicAllocation.shuffleTracking.timeout

(in millis)

Default: The largest value representable as an Int

sustainedSchedulerBacklogTimeout¶

spark.dynamicAllocation.sustainedSchedulerBacklogTimeout

Default: spark.dynamicAllocation.schedulerBacklogTimeout