getStatuses finds MapStatus.md[MapStatuses] for the input shuffleId in the <> internal cache and, when not available, fetches them from a remote MapOutputTrackerMaster.md[MapOutputTrackerMaster] (using RPC).
Internally, getStatuses first queries the <mapStatuses internal cache>> and returns the map outputs if found.
If not found (in the mapStatuses internal cache), you should see the following INFO message in the logs:
Don't have map outputs for shuffle [id], fetching them
If some other process fetches the map outputs for the shuffleId (as recorded in fetching internal registry), getStatuses waits until it is done.
When no other process fetches the map outputs, getStatuses registers the input shuffleId in fetching internal registry (of shuffle map outputs being fetched).
You should see the following INFO message in the logs:
Doing the fetch; tracker endpoint = [trackerEndpoint]
getStatuses sends a GetMapOutputStatuses RPC remote message for the input shuffleId to the trackerEndpoint expecting a Array[Byte].
NOTE: getStatuses requests shuffle map outputs remotely within a timeout and with retries. Refer to rpc:RpcEndpointRef.md[RpcEndpointRef].
getStatuses <> and records the result in the <mapStatuses internal cache>>.
You should see the following INFO message in the logs:
Got the output locations
getStatuses removes the input shuffleId from fetching internal registry.
You should see the following DEBUG message in the logs:
Fetching map output statuses for shuffle [id] took [time] ms
If getStatuses could not find the map output locations for the input shuffleId (locally and remotely), you should see the following ERROR message in the logs and throws a MetadataFetchFailedException.
Missing all output locations for shuffle [id]
NOTE: getStatuses is used when MapOutputTracker <> and <ShuffleDependency>>.
== [[logging]] Logging
Enable ALL logging level for org.apache.spark.MapOutputTrackerWorker logger to see what happens inside.