Skip to content

Spark Configuration Properties

spark.dynamicAllocation.cachedExecutorIdleTimeout

How long (in seconds) to keep blocks cached

Default: The largest value representable as an Int

Must be >= 0

Used when:

spark.dynamicAllocation.enabled

Default: false

Used when:

spark.dynamicAllocation.executorAllocationRatio

Default: 1.0

Must be between 0 (exclusive) and 1.0 (inclusive)

Used when:

  • ExecutorAllocationManager is created

spark.dynamicAllocation.executorIdleTimeout

Default: 60

spark.dynamicAllocation.initialExecutors

Default: spark.dynamicAllocation.minExecutors

spark.dynamicAllocation.maxExecutors

Default: The largest value representable as an Int

spark.dynamicAllocation.minExecutors

Default: 0

spark.dynamicAllocation.schedulerBacklogTimeout

(in seconds)

Default: 1

spark.dynamicAllocation.shuffleTracking.enabled

Default: false

Used when:

spark.dynamicAllocation.shuffleTracking.timeout

(in millis)

Default: The largest value representable as an Int

spark.dynamicAllocation.sustainedSchedulerBacklogTimeout

Default: spark.dynamicAllocation.schedulerBacklogTimeout


Last update: 2020-12-27