Skip to content

ShuffleStatus

ShuffleStatus is used by MapOutputTrackerMaster to keep track of the shuffle map outputs of a ShuffleMapStage.

Creating Instance

ShuffleStatus takes the following to be created:

ShuffleStatus is created when:

Registering Shuffle Map Output

addMapOutput(
  mapIndex: Int,
  status: MapStatus): Unit

addMapOutput...FIXME

addMapOutput is used when:

Deregistering Shuffle Map Output

removeMapOutput(
  mapIndex: Int,
  bmAddress: BlockManagerId): Unit

removeMapOutput...FIXME

removeMapOutput is used when:

Missing Partitions

findMissingPartitions(): Seq[Int]

findMissingPartitions...FIXME

findMissingPartitions is used when:

Serializing Shuffle Map Output Statuses

serializedMapStatus(
  broadcastManager: BroadcastManager,
  isLocal: Boolean,
  minBroadcastSize: Int,
  conf: SparkConf): Array[Byte]

serializedMapStatus...FIXME

serializedMapStatus is used when:

Logging

Enable ALL logging level for org.apache.spark.ShuffleStatus logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.ShuffleStatus=ALL

Refer to Logging.


Last update: 2021-04-30
Back to top