Skip to content

Spark Configuration Properties of External Shuffle Service

The following are configuration properties of External Shuffle Service.

spark.shuffle.service.db.enabled

Whether to use db in ExternalShuffleService. Note that this only affects standalone mode.

Default: true

Used when:

spark.shuffle.service.enabled

Controls whether to use the External Shuffle Service

Default: false

Note

LocalSparkCluster turns this property off explicitly when started.

Used when:

spark.shuffle.service.fetch.rdd.enabled

Enables ExternalShuffleService for fetching disk persisted RDD blocks.

When enabled with Dynamic Resource Allocation executors having only disk persisted blocks are considered idle after spark.dynamicAllocation.executorIdleTimeout and will be released accordingly.

Default: false

Used when:

spark.shuffle.service.port

Port of the external shuffle service

Default: 7337

Used when:


Last update: 2020-12-26
Back to top