Skip to content

MapOutputTrackerMasterEndpoint

MapOutputTrackerMasterEndpoint is an RpcEndpoint for MapOutputTrackerMaster.

MapOutputTrackerMasterEndpoint is registered under the name of MapOutputTracker (on the driver).

Creating Instance

MapOutputTrackerMasterEndpoint takes the following to be created:

MapOutputTrackerMasterEndpoint is created when:

  • SparkEnv is created (for the driver and executors)

While being created, MapOutputTrackerMasterEndpoint prints out the following DEBUG message to the logs:

init

Messages

GetMapOutputStatuses

GetMapOutputStatuses(
  shuffleId: Int)

Posted when MapOutputTrackerWorker is requested for shuffle map outputs for a given shuffle ID

When received, MapOutputTrackerMasterEndpoint prints out the following INFO message to the logs:

Asked to send map output locations for shuffle [shuffleId] to [hostPort]

In the end, MapOutputTrackerMasterEndpoint requests the MapOutputTrackerMaster to post a GetMapOutputMessage (with the input shuffleId). Whatever is returned from MapOutputTrackerMaster becomes the response.

StopMapOutputTracker

Posted when MapOutputTrackerMaster is requested to stop.

When received, MapOutputTrackerMasterEndpoint prints out the following INFO message to the logs:

MapOutputTrackerMasterEndpoint stopped!

MapOutputTrackerMasterEndpoint confirms the request (by replying true) and stops.

Logging

Enable ALL logging level for org.apache.spark.MapOutputTrackerMasterEndpoint logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.MapOutputTrackerMasterEndpoint=ALL

Refer to Logging.


Last update: 2020-12-10