AccumulatorV2¶
AccumulatorV2[IN, OUT]
is an abstraction of accumulators
AccumulatorV2
is a Java Serializable.
Contract¶
Adding Value¶
add(
v: IN): Unit
Accumulates (adds) the given v
value to this accumulator
Copying Accumulator¶
copy(): AccumulatorV2[IN, OUT]
Is Zero Value¶
isZero: Boolean
Merging Updates¶
merge(
other: AccumulatorV2[IN, OUT]): Unit
Resetting Accumulator¶
reset(): Unit
Value¶
value: OUT
The current value of this accumulator
Used when:
TaskRunner
is requested to collectAccumulatorsAndResetStatusOnFailureAccumulatorSource
is requested to registerDAGScheduler
is requested to update accumulatorsTaskSchedulerImpl
is requested to executorHeartbeatReceivedTaskSetManager
is requested to handleSuccessfulTaskJsonProtocol
is requested to taskEndReasonFromJson- others
Implementations¶
- AggregatingAccumulator (Spark SQL)
- CollectionAccumulator
- DoubleAccumulator
- EventTimeStatsAccum (Spark Structured Streaming)
- LongAccumulator
- SetAccumulator (Spark SQL)
- SQLMetric (Spark SQL)
Converting this Accumulator to AccumulableInfo¶
toInfo(
update: Option[Any],
value: Option[Any]): AccumulableInfo
toInfo
determines whether the accumulator is internal based on the name (and whether it uses the internal.metrics prefix) and uses it to create an AccumulableInfo.
toInfo
is used when:
TaskRunner
is requested to collectAccumulatorsAndResetStatusOnFailureDAGScheduler
is requested to updateAccumulatorsTaskSchedulerImpl
is requested to executorHeartbeatReceivedJsonProtocol
is requested to taskEndReasonFromJsonSQLAppStatusListener
(Spark SQL) is requested to handle aSparkListenerTaskEnd
event (onTaskEnd
)
Registering Accumulator¶
register(
sc: SparkContext,
name: Option[String] = None,
countFailedValues: Boolean = false): Unit
register
...FIXME
register
is used when:
SparkContext
is requested to register an accumulatorTaskMetrics
is requested to register task accumulatorsCollectMetricsExec
(Spark SQL) is requested for anAggregatingAccumulator
SQLMetrics
(Spark SQL) is used to create a performance metric
writeReplace¶
writeReplace(): Any
writeReplace
...FIXME
Note
writeReplace
is part of the java.io.Serializable abstraction.
Quote
Serializable classes that need to designate an alternative object to be used when writing an object to the stream should implement this special method with the exact signature:
ANY-ACCESS-MODIFIER Object writeReplace() throws ObjectStreamException;