Skip to content

TaskMetrics

TaskMetrics is a collection of metrics (accumulators) tracked during execution of a task.

Creating Instance

TaskMetrics takes no input arguments to be created.

TaskMetrics is created when:

Metrics

Memory Bytes Spilled

memoryBytesSpilled metric

memoryBytesSpilled is exposed using Dropwizard metrics system using ExecutorSource as memoryBytesSpilled.

incMemoryBytesSpilled is used when:

TaskContext

TaskMetrics is available using TaskContext.taskMetrics.

TaskContext.get.taskMetrics

Serializable

TaskMetrics is a Serializable (Java).

Task

TaskMetrics is part of Task.

task.metrics

SparkListener

TaskMetrics is available using SparkListener and intercepting SparkListenerTaskEnd events.

StatsReportListener

StatsReportListener can be used for summary statistics at runtime (after a stage completes).

Spark History Server

Spark History Server uses EventLoggingListener to intercept post-execution statistics (incl. TaskMetrics).


Last update: 2021-08-02
Back to top