Skip to content

MetricsReporter

MetricsReporter is a metrics Source (Spark Core) to report metrics of a streaming query.

MetricsReporter uses the last StreamingQueryProgress (of the StreamExecution) if available or simply defaults to a "zero" value.

spark.sql.streaming.metricsEnabled

MetricsReporter is registered when spark.sql.streaming.metricsEnabled is enabled.

Creating Instance

MetricsReporter takes the following to be created:

MetricsReporter is created when:

Source Name

MetricsReporter is given a source name when created.

The source name is of the format (with the name if available or id):

spark.streaming.[name or id]

Gauges

inputRate-total

Reports inputRowsPerSecond (across all streaming sources)

processingRate-total

Reports processedRowsPerSecond (across all streaming sources)

latency

Reports triggerExecution duration of the last StreamingQueryProgress

eventTime-watermark

Reports watermark of the last StreamingQueryProgress

Format: yyyy-MM-dd'T'HH:mm:ss.SSS'Z'

states-rowsTotal

Reports the total of numRowsTotal of all StateOperatorProgresses of the last StreamingQueryProgress

states-usedBytes

Reports the total of memoryUsedBytes of all StateOperatorProgresses of the last StreamingQueryProgress