Skip to content


ShuffleWriter of K keys and V values (ShuffleWriter[K, V]) is an abstraction of <> that can <> (of a RDD partition) to a shuffle system.

ShuffleWriter is used when[ShuffleMapTask] is requested to[run].

== [[implementations]] ShuffleWriters

.ShuffleWriters [cols="40m,60",options="header",width="100%"] |=== | ShuffleWriter | Description

|[BypassMergeSortShuffleWriter] | [[BypassMergeSortShuffleWriter]] ShuffleWriter for a[BypassMergeSortShuffleHandle]

|[SortShuffleWriter] | [[SortShuffleWriter]] Fallback ShuffleWriter (when neither <> nor <> could be used)

|[UnsafeShuffleWriter] | [[UnsafeShuffleWriter]] ShuffleWriter for[SerializedShuffleHandles]


== [[stop]] Stopping ShuffleWriter

[source, scala]

stop( success: Boolean): Option[MapStatus]

Stops (closes) the ShuffleWriter and returns a[MapStatus] if the writing completed successfully. The success flag is the status of the task execution.

stop is used when ShuffleMapTask is requested to[run].

== [[write]] Writing Partition Records Out to Shuffle System

[source, scala]

write( records: Iterator[Product2[K, V]]): Unit

Writes key-value records out to a shuffle system.

write is used when ShuffleMapTask is requested to[run].

Last update: 2020-10-09