MetricsReporter¶

MetricsReporter is a metrics Source (Spark Core) to report metrics of a streaming query.

MetricsReporter uses the last StreamingQueryProgress (of the StreamExecution) if available or simply defaults to a "zero" value.

Creating Instance¶

MetricsReporter takes the following to be created:

• StreamExecution
• Source Name

MetricsReporter is created when:

• StreamExecution is requested for the MetricsReporter

Source Name¶

MetricsReporter is given a source name when created.

The source name is of the format (with the name if available or id):

spark.streaming.[name or id]

Gauges¶

inputRate-total¶

Reports inputRowsPerSecond (across all streaming sources)

processingRate-total¶

Reports processedRowsPerSecond (across all streaming sources)

latency¶

Reports triggerExecution duration of the last StreamingQueryProgress

eventTime-watermark¶

Reports watermark of the last StreamingQueryProgress

Format: yyyy-MM-dd'T'HH:mm:ss.SSS'Z'

states-rowsTotal¶

Reports the total of numRowsTotal of all StateOperatorProgresses of the last StreamingQueryProgress

states-usedBytes¶

Reports the total of memoryUsedBytes of all StateOperatorProgresses of the last StreamingQueryProgress