Skip to content


MapStatus is an abstraction of shuffle map output statuses with an estimated size, location and map Id.

MapStatus is a result of executing a ShuffleMapTask.

After a ShuffleMapTask has finished execution successfully, DAGScheduler is requested to handle a ShuffleMapTask completion that in turn requests the MapOutputTrackerMaster to register the MapStatus.


Estimated Size

  reduceId: Int): Long

Estimated size (in bytes)

Used when:


location: BlockManagerId

BlockManagerId of the shuffle map output (i.e. the BlockManager where a ShuffleMapTask ran and the result is stored)

Used when:

Map Id

mapId: Long

Map Id of the shuffle map output

Used when:


Sealed Trait

MapStatus is a Scala sealed trait which means that all of the implementations are in the same compilation unit (a single file).


MapStatus utility uses spark.shuffle.minNumPartitionsToHighlyCompress internal configuration property for the minimum number of partitions to prefer a HighlyCompressedMapStatus.

Creating MapStatus

  loc: BlockManagerId,
  uncompressedSizes: Array[Long],
  mapTaskId: Long): MapStatus

apply creates a HighlyCompressedMapStatus when the number of uncompressedSizes is above minPartitionsToUseHighlyCompressMapStatus threshold. Otherwise, apply creates a CompressedMapStatus.

apply is used when:

Back to top