MapOutputTrackerMasterEndpoint

MapOutputTrackerMasterEndpoint is a RpcEndpoint for MapOutputTrackerMaster.

MapOutputTrackerMasterEndpoint handles the following messages:

Enable INFO or DEBUG logging levels for org.apache.spark.MapOutputTrackerMasterEndpoint logger to see what happens in MapOutputTrackerMasterEndpoint.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.MapOutputTrackerMasterEndpoint=DEBUG

Refer to Logging.

Creating MapOutputTrackerMasterEndpoint Instance

MapOutputTrackerMasterEndpoint takes the following when created:

  1. rpcEnv — RpcEnv

  2. tracker — MapOutputTrackerMaster

  3. conf — SparkConf

When created, you should see the following DEBUG message in the logs:

DEBUG init
MapOutputTrackerMasterEndpoint is created when SparkEnv is created for the driver and executors.

GetMapOutputStatuses Message

GetMapOutputStatuses(shuffleId: Int)
extends MapOutputTrackerMessage

When GetMapOutputStatuses arrives, MapOutputTrackerMasterEndpoint reads the host and the port of the sender.

You should see the following INFO message in the logs:

INFO Asked to send map output locations for shuffle [shuffleId] to [hostPort]

MapOutputTrackerMasterEndpoint posts a GetMapOutputMessage to MapOutputTrackerMaster (with shuffleId and the current RpcCallContext).

GetMapOutputStatuses is posted when MapOutputTracker fetches shuffle map outputs remotely.

StopMapOutputTracker Message

StopMapOutputTracker
extends MapOutputTrackerMessage

When StopMapOutputTracker arrives, you should see the following INFO message in the logs:

INFO MapOutputTrackerMasterEndpoint stopped!

MapOutputTrackerMasterEndpoint confirms the request (by replying true) and stops itself (and stops accepting messages).

StopMapOutputTracker is posted when MapOutputTrackerMaster stops.