ShuffleWriter

ShuffleWriter of K keys and V values (ShuffleWriter[K, V]) is an abstraction of shuffle writers that can write key-value records (of a RDD partition) to a shuffle system.

ShuffleWriter is used when ShuffleMapTask is requested to run.

ShuffleWriters

Table 1. ShuffleWriters
ShuffleWriter Description

BypassMergeSortShuffleWriter

ShuffleWriter for a BypassMergeSortShuffleHandle

SortShuffleWriter

Fallback ShuffleWriter (when neither BypassMergeSortShuffleWriter nor UnsafeShuffleWriter could be used)

UnsafeShuffleWriter

ShuffleWriter for SerializedShuffleHandles

Stopping ShuffleWriter

stop(
  success: Boolean): Option[MapStatus]

Stops (closes) the ShuffleWriter and returns a MapStatus if the writing completed successfully. The success flag is the status of the task execution.

stop is used when ShuffleMapTask is requested to run.

Writing Partition Records Out to Shuffle System

write(
  records: Iterator[Product2[K, V]]): Unit

Writes key-value records out to a shuffle system.

write is used when ShuffleMapTask is requested to run.