SQLMetric¶

SQLMetric is an AccumulatorV2 (Spark Core) for performance metrics of physical operators.

Creating Instance¶

SQLMetric takes the following to be created:

• Metric Type
• Initial Value

SQLMetric is created using the specialized utilities:

• createMetric
• createSizeMetric
• createTimingMetric
• createNanoTimingMetric
• createAverageMetric

createAverageMetric¶

createAverageMetric(
  sc: SparkContext,
  name: String): SQLMetric

createAverageMetric creates a SQLMetric with average type and registers it with the given name.

createMetric¶

createMetric(
  sc: SparkContext,
  name: String): SQLMetric

createMetric creates a SQLMetric with sum type and registers it with the given name.

createNanoTimingMetric¶

createNanoTimingMetric(
  sc: SparkContext,
  name: String): SQLMetric

createNanoTimingMetric creates a SQLMetric with nsTiming type and registers it with the given name.

createSizeMetric¶

createSizeMetric(
  sc: SparkContext,
  name: String): SQLMetric

createSizeMetric creates a SQLMetric with size type and registers it with the given name.

createTimingMetric¶

createTimingMetric(
  sc: SparkContext,
  name: String): SQLMetric

createTimingMetric creates a SQLMetric with timing type and registers it with the given name.

Metric Types¶

SQLMetric is given a metric type to be created:

• average
• nsTiming
• size
• sum
• timing

Posting Driver-Side Metric Updates¶

postDriverMetricUpdates(
  sc: SparkContext,
  executionId: String,
  metrics: Seq[SQLMetric]): Unit

postDriverMetricUpdates posts a SparkListenerDriverAccumUpdates event to LiveListenerBus (only if executionId is specified).

postDriverMetricUpdates is used when:

• BasicWriteJobStatsTracker is requested for processStats
• BroadcastExchangeExec is requested for relationFuture
• FileSourceScanExec physical operator is requested for sendDriverMetrics
• SubqueryBroadcastExec physical operator is requested for relationFuture
• SubqueryExec physical operator is requested for relationFuture