Skip to content


ShuffleWriteProcessor is used by ShuffleMapTask to write partition records to the shuffle system.

Writing Partition Records to Shuffle System

  rdd: RDD[_],
  dep: ShuffleDependency[_, _, _],
  mapId: Long,
  context: TaskContext,
  partition: Partition): MapStatus

write requests the ShuffleManager for the ShuffleWriter for the ShuffleHandle (of the given ShuffleDependency).

write requests the ShuffleWriter to write out records (of the given Partition and RDD).

In the end, write requests the ShuffleWriter to stop (with the success flag enabled).

In case of any Exceptions, write requests the ShuffleWriter to stop (with the success flag disabled).

write is used when ShuffleMapTask is requested to run.

Creating MetricsReporter

  context: TaskContext): ShuffleWriteMetricsReporter

createMetricsReporter creates a ShuffleWriteMetricsReporter from the given TaskContext.

createMetricsReporter requests the given TaskContext for TaskMetrics and then for the ShuffleWriteMetrics.

Back to top