Skip to content


MetricsConfig is the configuration of the MetricsSystem (i.e. metrics sources and sinks).

MetricsConfig is <> when[MetricsSystem] is.

MetricsConfig uses as the default metrics configuration file. It is configured using[spark.metrics.conf] configuration property. The file is first loaded from the path directly before using Spark's CLASSPATH.

MetricsConfig accepts a metrics configuration using spark.metrics.conf.-prefixed configuration properties.

Spark comes with conf/ file that is a template of metrics configuration.

MetricsConfig <> that the <> are always defined.

[[default-properties]] .MetricsConfig's Default Metrics Properties [cols="1,2",options="header",width="100%"] |=== | Name | Description

| *.sink.servlet.class | org.apache.spark.metrics.sink.MetricsServlet

| *.sink.servlet.path | /metrics/json

| master.sink.servlet.path | /metrics/master/json

| applications.sink.servlet.path | /metrics/applications/json |===


The order of precedence of metrics configuration settings is as follows:

. <> .[spark.metrics.conf] configuration property or configuration file . spark.metrics.conf.-prefixed Spark properties ====

[[creating-instance]] [[conf]] MetricsConfig takes a[SparkConf] when created.

[[internal-registries]] .MetricsConfig's Internal Registries and Counters [cols="1,2",options="header",width="100%"] |=== | Name | Description

| [[properties]] properties |[java.util.Properties] with metrics properties

Used to <> per-subsystem's <>.

| [[perInstanceSubProperties]] perInstanceSubProperties | Lookup table of metrics properties per subsystem |===

=== [[initialize]] Initializing MetricsConfig -- initialize Method

[source, scala]

initialize(): Unit

initialize <> and <> (that is defined using[spark.metrics.conf] configuration property).

initialize takes all Spark properties that start with spark.metrics.conf. prefix from <> and adds them to <> (without the prefix).

In the end, initialize splits <> with the default configuration (denoted as *) assigned to all subsystems afterwards.

NOTE: initialize accepts * (star) for the default configuration or any combination of lower- and upper-case letters for Spark subsystem names.

NOTE: initialize is used exclusively when MetricsSystem is[created].

=== [[setDefaultProperties]] setDefaultProperties Internal Method

[source, scala]

setDefaultProperties(prop: Properties): Unit

setDefaultProperties sets the <> (in the input prop).

NOTE: setDefaultProperties is used exclusively when MetricsConfig <>.

=== [[loadPropertiesFromFile]] Loading Custom Metrics Configuration File or -- loadPropertiesFromFile Method

[source, scala]

loadPropertiesFromFile(path: Option[String]): Unit

loadPropertiesFromFile tries to open the input path file (if defined) or the default metrics configuration file (on CLASSPATH).

If either file is available, loadPropertiesFromFile loads the properties (to <> registry).

In case of exceptions, you should see the following ERROR message in the logs followed by the exception.

ERROR Error loading configuration file [file]

NOTE: loadPropertiesFromFile is used exclusively when MetricsConfig <>.

=== [[subProperties]] Grouping Properties Per Subsystem -- subProperties Method

[source, scala]

subProperties(prop: Properties, regex: Regex): mutable.HashMap[String, Properties]

subProperties takes prop properties and destructures keys given regex. subProperties takes the matching prefix (of a key per regex) and uses it as a new key with the value(s) being the matching suffix(es).

[source, scala] => (driver, (

NOTE: subProperties is used when MetricsConfig <> (to apply the default metrics configuration) and when MetricsSystem[registers metrics sources] and[sinks].

=== [[getInstance]] getInstance Method

[source, scala]

getInstance(inst: String): Properties


NOTE: getInstance is used when...FIXME

Last update: 2020-10-06