Skip to content


ShuffleWriteProcessor controls write behavior in ShuffleMapTasks while writing partition records out to the shuffle system.

ShuffleWriteProcessor is used to create a ShuffleDependency.

Creating Instance

ShuffleWriteProcessor takes no arguments to be created.

ShuffleWriteProcessor is created when:

  • ShuffleDependency is created
  • ShuffleExchangeExec (Spark SQL) physical operator is requested to createShuffleWriteProcessor

Writing Partition Records to Shuffle System

  rdd: RDD[_],
  dep: ShuffleDependency[_, _, _],
  mapId: Long,
  context: TaskContext,
  partition: Partition): MapStatus

write requests the ShuffleManager for the ShuffleWriter for the ShuffleHandle (of the given ShuffleDependency).

write requests the ShuffleWriter to write out records (of the given Partition and RDD).

In the end, write requests the ShuffleWriter to stop (with the success flag enabled).

In case of any Exceptions, write requests the ShuffleWriter to stop (with the success flag disabled).

write is used when ShuffleMapTask is requested to run.

Creating MetricsReporter

  context: TaskContext): ShuffleWriteMetricsReporter

createMetricsReporter creates a ShuffleWriteMetricsReporter from the given TaskContext.

createMetricsReporter requests the given TaskContext for TaskMetrics and then for the ShuffleWriteMetrics.