ShuffleMetricsSource is the metrics source of a BlockManager for shuffle-related metrics.

ShuffleMetricsSource lives on a Spark executor and is registered only when a Spark application runs in a non-local / cluster mode.

spark ShuffleMetricsSource
Figure 1. Registering ShuffleMetricsSource with "executor" MetricsSystem

ShuffleMetricsSource is created exclusively when BlockManager is requested for the shuffleMetricsSource.

When created, ShuffleMetricsSource gets a MetricSet that BlockManager requests from the ShuffleClient (only when in a non-local / cluster mode).

ShuffleMetricsSource is registered under the following source name per the type of a BlockManager:

Table 1. ShuffleMetricsSource’s Gauge Metrics (in alphabetical order)
Name Type Description
Since Executor does not have a web UI attached you cannot access the metrics using the HTTP protocol (through MetricsServlet JSON metrics sink).

Creating ShuffleMetricsSource Instance

ShuffleMetricsSource takes the following when created: