MapOutputTrackerMasterEndpoint

MapOutputTrackerMasterEndpoint is a RpcEndpoint for MapOutputTrackerMaster to handle the following messages:

Creating Instance

MapOutputTrackerMasterEndpoint takes the following to be created:

While being created, MapOutputTrackerMasterEndpoint prints out the following DEBUG message to the logs:

init

Messages

GetMapOutputStatuses

GetMapOutputStatuses(shuffleId: Int)

When received, MapOutputTrackerMasterEndpoint prints out the following INFO message to the logs:

Asked to send map output locations for shuffle [shuffleId] to [hostPort]

MapOutputTrackerMasterEndpoint requests the MapOutputTrackerMaster to post a GetMapOutputMessage.

GetMapOutputStatuses is posted when MapOutputTrackerWorker is requested for shuffle map outputs for a given shuffle ID.

StopMapOutputTracker

StopMapOutputTracker

When StopMapOutputTracker arrives, you should see the following INFO message in the logs:

INFO MapOutputTrackerMasterEndpoint stopped!

MapOutputTrackerMasterEndpoint confirms the request (by replying true) and stops itself (and stops accepting messages).

StopMapOutputTracker is posted when MapOutputTrackerMaster is requested to stop.

Logging

Enable ALL logging level for org.apache.spark.MapOutputTrackerMasterEndpoint logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.MapOutputTrackerMasterEndpoint=ALL

Refer to Logging.