SpillListener — Detecting Spills in Jobs (for Testing)

SpillListener is a SparkListener that intercepts (listens to) the following events:

SpillListener is created exclusively for TestUtils when requested for assertSpilled and assertNotSpilled.

SpillListener takes no input arguments to be created.

SpillListener is a Scala private class.
Table 1. SpillListener’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description


(mutable.HashMap[Int, ArrayBuffer[TaskMetrics]])

Used when…​FIXME



Used when…​FIXME

onTaskEnd Callback

onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit
onTaskEnd is part of the SparkListener Contract to…​FIXME.


onStageCompleted Callback

onStageCompleted(stageComplete: SparkListenerStageCompleted): Unit
onStageCompleted is part of the SparkListener Contract to…​FIXME.