Shuffle System¶

Shuffle System is a core service of Apache Spark that is responsible for shuffle blocks.

The main core abstraction is ShuffleManager with SortShuffleManager as the default and only known implementation.

spark.shuffle.manager configuration property allows for a custom ShuffleManager.

Shuffle System uses shuffle handles, readers and writers.

Resources¶

• Improving Apache Spark Downscaling by Christopher Crosbie (Google) Ben Sidhom (Google)
• Spark shuffle introduction by Raymond Liu (aka colorant)