MapOutputTrackerMasterEndpoint¶

MapOutputTrackerMasterEndpoint is an RpcEndpoint for MapOutputTrackerMaster.

MapOutputTrackerMasterEndpoint is registered under the name of MapOutputTracker (on the driver).

Creating Instance¶

MapOutputTrackerMasterEndpoint takes the following to be created:

• RpcEnv
• MapOutputTrackerMaster
• SparkConf

MapOutputTrackerMasterEndpoint is created when:

• SparkEnv is created (for the driver and executors)

While being created, MapOutputTrackerMasterEndpoint prints out the following DEBUG message to the logs:

init

Messages¶

GetMapOutputStatuses¶

GetMapOutputStatuses(
  shuffleId: Int)

Posted when MapOutputTrackerWorker is requested for shuffle map outputs for a given shuffle ID

When received, MapOutputTrackerMasterEndpoint prints out the following INFO message to the logs:

Asked to send map output locations for shuffle [shuffleId] to [hostPort]

In the end, MapOutputTrackerMasterEndpoint requests the MapOutputTrackerMaster to post a GetMapOutputMessage (with the input shuffleId). Whatever is returned from MapOutputTrackerMaster becomes the response.

StopMapOutputTracker¶

Posted when MapOutputTrackerMaster is requested to stop.

When received, MapOutputTrackerMasterEndpoint prints out the following INFO message to the logs:

MapOutputTrackerMasterEndpoint stopped!

MapOutputTrackerMasterEndpoint confirms the request (by replying true) and stops.

Logging¶

Enable ALL logging level for org.apache.spark.MapOutputTrackerMasterEndpoint logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.MapOutputTrackerMasterEndpoint=ALL

Refer to Logging.