Skip to content

ShuffleWriter

ShuffleWriter[K, V] (of K keys and V values) is an abstraction of shuffle writers that can write out key-value records (of a RDD partition) to a shuffle system.

ShuffleWriter is used when ShuffleMapTask is requested to run (and uses a ShuffleWriteProcessor to write partition records to a shuffle system).

Contract

Writing Out Partition Records to Shuffle System

write(
  records: Iterator[Product2[K, V]]): Unit

Writes key-value records (of a partition) out to a shuffle system

Used when:

  • ShuffleWriteProcessor is requested to write

Stopping ShuffleWriter

stop(
  success: Boolean): Option[MapStatus]

Stops (closes) the ShuffleWriter and returns a MapStatus if the writing completed successfully. The success flag is the status of the task execution.

Used when:

  • ShuffleWriteProcessor is requested to write

Implementations


Last update: 2020-11-23