AccumulatorV2¶

AccumulatorV2[IN, OUT] is an abstraction of accumulators

AccumulatorV2 is a Java Serializable.

Contract¶

Adding Value¶

add(
  v: IN): Unit

Accumulates (adds) the given v value to this accumulator

Copying Accumulator¶

copy(): AccumulatorV2[IN, OUT]

Is Zero Value¶

isZero: Boolean

Merging Updates¶

merge(
  other: AccumulatorV2[IN, OUT]): Unit

Resetting Accumulator¶

reset(): Unit

Value¶

value: OUT

The current value of this accumulator

Used when:

• TaskRunner is requested to collectAccumulatorsAndResetStatusOnFailure
• AccumulatorSource is requested to register
• DAGScheduler is requested to update accumulators
• TaskSchedulerImpl is requested to executorHeartbeatReceived
• TaskSetManager is requested to handleSuccessfulTask
• JsonProtocol is requested to taskEndReasonFromJson
• others

Implementations¶

• AggregatingAccumulator (Spark SQL)
• CollectionAccumulator
• DoubleAccumulator
• EventTimeStatsAccum (Spark Structured Streaming)
• LongAccumulator
• SetAccumulator (Spark SQL)
• SQLMetric (Spark SQL)

Converting this Accumulator to AccumulableInfo¶

toInfo(
  update: Option[Any],
  value: Option[Any]): AccumulableInfo

toInfo determines whether the accumulator is internal based on the name (and whether it uses the internal.metrics prefix) and uses it to create an AccumulableInfo.

toInfo is used when:

• TaskRunner is requested to collectAccumulatorsAndResetStatusOnFailure
• DAGScheduler is requested to updateAccumulators
• TaskSchedulerImpl is requested to executorHeartbeatReceived
• JsonProtocol is requested to taskEndReasonFromJson
• SQLAppStatusListener (Spark SQL) is requested to handle a SparkListenerTaskEnd event (onTaskEnd)

Registering Accumulator¶

register(
  sc: SparkContext,
  name: Option[String] = None,
  countFailedValues: Boolean = false): Unit

register...FIXME

register is used when:

• SparkContext is requested to register an accumulator
• TaskMetrics is requested to register task accumulators
• CollectMetricsExec (Spark SQL) is requested for an AggregatingAccumulator
• SQLMetrics (Spark SQL) is used to create a performance metric

Serializing AccumulatorV2¶

writeReplace(): Any

writeReplace is part of the Serializable (Java) abstraction (to designate an alternative object to be used when writing an object to the stream).

writeReplace...FIXME

Deserializing AccumulatorV2¶

readObject(
  in: ObjectInputStream): Unit

readObject is part of the Serializable (Java) abstraction (for special handling during deserialization).

readObject reads the non-static and non-transient fields of the AccumulatorV2 from the given ObjectInputStream.

If the atDriverSide internal flag is turned on, readObject turns it off (to indicate readObject is executed on an executor). Otherwise, atDriverSide internal flag is turned on.

readObject requests the active TaskContext to register this accumulator.