ShuffleStatus
= [[ShuffleStatus]] ShuffleStatus
ShuffleStatus is a registry of shuffle map outputs (of a shuffle stage).
ShuffleStatus is managed by scheduler:MapOutputTrackerMaster.md#shuffleStatuses[MapOutputTrackerMaster] to keep track of shuffle map outputs across shuffle stages.
== [[creating-instance]][[numPartitions]] Creating Instance
ShuffleStatus takes a single number of partitions to be created.
== [[addMapOutput]] Registering Shuffle Map Output
[source, scala]¶
addMapOutput( mapId: Int, status: MapStatus): Unit
addMapOutput...FIXME
addMapOutput is used when MapOutputTrackerMaster is requested to scheduler:MapOutputTrackerMaster.md#registerMapOutput[register a shuffle map output].
== [[removeMapOutput]] Deregistering Shuffle Map Output
[source, scala]¶
removeMapOutput( mapId: Int, bmAddress: BlockManagerId): Unit
removeMapOutput...FIXME
removeMapOutput is used when MapOutputTrackerMaster is requested to scheduler:MapOutputTrackerMaster.md#unregisterMapOutput[unregister a shuffle map output].
== [[serializedMapStatus]] Serializing Shuffle Map Output Statuses
[source, scala]¶
serializedMapStatus( broadcastManager: BroadcastManager, isLocal: Boolean, minBroadcastSize: Int): Array[Byte]
serializedMapStatus...FIXME
serializedMapStatus is used when MapOutputTrackerMaster is requested to scheduler:MapOutputTrackerMaster.md#run[send the map output locations of a shuffle] (on the MessageLoop dispatcher thread).
== [[findMissingPartitions]] Finding Missing Partitions
[source, scala]¶
findMissingPartitions(): Seq[Int]¶
findMissingPartitions...FIXME
findMissingPartitions is used when MapOutputTrackerMaster is requested for scheduler:MapOutputTrackerMaster.md#findMissingPartitions[missing partitions (that need to be computed)].
== [[invalidateSerializedMapOutputStatusCache]] Invalidating Serialized Map Output Status Cache
[source, scala]¶
invalidateSerializedMapOutputStatusCache(): Unit¶
invalidateSerializedMapOutputStatusCache...FIXME
invalidateSerializedMapOutputStatusCache is used when:
-
ShuffleStatus is requested to <
>, < >, < > -
MapOutputTrackerMaster is requested to scheduler:MapOutputTrackerMaster.md#unregisterShuffle[unregister a shuffle]
== [[removeOutputsByFilter]] Deregistering Shuffle Map Outputs by Filter
[source, scala]¶
removeOutputsByFilter( f: (BlockManagerId) => Boolean): Unit
removeOutputsByFilter...FIXME
removeOutputsByFilter is used when:
-
ShuffleStatus is requested to <
>, < > -
MapOutputTrackerMaster is requested to scheduler:MapOutputTrackerMaster.md#unregisterAllMapOutput[unregister all map outputs of a given shuffle stage]
== [[removeOutputsOnExecutor]] Deregistering Shuffle Map Outputs Associated with Executor
[source, scala]¶
removeOutputsOnExecutor( execId: String): Unit
removeOutputsOnExecutor...FIXME
removeOutputsOnExecutor is used when MapOutputTrackerMaster is requested to scheduler:MapOutputTrackerMaster.md#removeOutputsOnExecutor[delete shuffle outputs associated with an executor].
== [[removeOutputsOnHost]] Deregistering Shuffle Map Outputs Associated with Host
[source, scala]¶
removeOutputsOnHost( host: String): Unit
removeOutputsOnHost...FIXME
removeOutputsOnHost is used when MapOutputTrackerMaster is requested to scheduler:MapOutputTrackerMaster.md#removeOutputsOnHost[delete shuffle outputs associated with a host].