EventTimeStatsAccum is an
AccumulatorV2 (Spark Core) that accumulates
Long values and produces an EventTimeStats.
EventTimeStatsAccum takes the following to be created:
- EventTimeStats (default: EventTimeStats.zero)
EventTimeStatsAccum is created when:
- EventTimeWatermarkExec unary physical operator is created (and initializes eventTimeStats)
EventTimeWatermarkExec Physical Operator
EventTimeWatermarkExec physical operator is requested to execute, every task simply adds the values of the event-time watermark column expression to the EventTimeStatsAccum accumulator.
As per design of Spark accumulators in Apache Spark, accumulator updates are automatically sent out (propagated) from tasks to the driver every heartbeat and then they are accumulated together.
EventTimeStatsAccum is given an EventTimeStats when created.
AccumulatorV2 methods are called,
EventTimeStatsAccum simply delegates them to the
EventTimeStats (that is responsible for event-time statistics, i.e. max, min, avg, count).
add( v: Long): Unit
add is part of the
AccumulatorV2 (Spark Core) abstraction.
add simply requests the EventTimeStats to add the given
add is used when EventTimeWatermarkExec physical operator is executed.