Spillable uses spark.shuffle.spill.numElementsForceSpillThreshold configuration property to force spilling in-memory objects to disk when requested to maybeSpill.
Spillable uses a threshold for the memory size (in bytes) to know when to spill to disk.
When the size of the in-memory collection is above the threshold, Spillable will try to acquire more memory. Unless given all requested memory, Spillable spills to disk.
The memory threshold starts as spark.shuffle.spill.initialMemoryThreshold configuration property and is increased every time Spillable is requested to spill to disk if needed, but managed to acquire required memory. The threshold goes back to the initial value when requested to release all memory.
forceSpill forcefully spills the Spillable to disk to release memory
forceSpill is used when Spillable is requested to spill an in-memory collection to disk.
maybeSpill( collection: C, currentMemory: Long): Boolean
maybeSpill is used when:
ExternalAppendOnlyMap is requested to insertAll
ExternalSorter is requested to attempt to spill an in-memory collection to disk if needed