Skip to content

EventTimeStatsAccum Accumulator

EventTimeStatsAccum is an AccumulatorV2 (Spark Core) that accumulates Long values and produces an EventTimeStats.

AccumulatorV2[Long, EventTimeStats]

Creating Instance

EventTimeStatsAccum takes the following to be created:

EventTimeStatsAccum is created when:

EventTimeWatermarkExec Physical Operator

When EventTimeWatermarkExec physical operator is requested to execute, every task simply adds the values of the event-time watermark column expression to the EventTimeStatsAccum accumulator.

As per design of Spark accumulators in Apache Spark, accumulator updates are automatically sent out (propagated) from tasks to the driver every heartbeat and then they are accumulated together.

EventTimeStats

EventTimeStatsAccum is given an EventTimeStats when created.

Every time AccumulatorV2 methods are called, EventTimeStatsAccum simply delegates them to the EventTimeStats (that is responsible for event-time statistics, i.e. max, min, avg, count).

Adding Value

add(
  v: Long): Unit

add is part of the AccumulatorV2 (Spark Core) abstraction.

add simply requests the EventTimeStats to add the given v value.

add is used when EventTimeWatermarkExec physical operator is executed.