CollectMetrics Logical Operator¶

CollectMetrics is a unary logical operator that represents Dataset.observe operator (in the logical query plan).

Creating Instance¶

CollectMetrics takes the following to be created:

Name
Metric NamedExpressions
Child logical operator

CollectMetrics is created when Dataset.observe operator is used.

Execution Planning¶

CollectMetrics is planned as CollectMetricsExec physical operator.

Demo¶

// TIP: Use AggregateExpressions
val q = spark
  .range(0, 5, 1, numPartitions = 1)
  .observe(name = "myMetric", expr = count('id))
val plan = q.queryExecution.logical

scala> println(plan.numberedTreeString)
00 'CollectMetrics myMetric, [count('id) AS count(id)#46]
01 +- Range (0, 5, step=1, splits=Some(1))