Skip to content

Shuffle System

Shuffle System is a core service of Apache Spark that is responsible for shuffle blocks.

The main core abstraction is ShuffleManager with SortShuffleManager as the default and only known implementation.

spark.shuffle.manager configuration property allows for a custom ShuffleManager.

Shuffle System uses shuffle handles, readers and writers.