MemoryConsumer is an abstraction of memory consumers (of TaskMemoryManager) that support spilling.
MemoryConsumers correspond to individual operators and data structures within a task.
TaskMemoryManager receives memory allocation requests from
MemoryConsumers and issues callbacks to consumers in order to trigger spilling when running low on memory.
MemoryConsumer basically tracks how much memory is allocated.
void spill() // (1) long spill( long size, MemoryConsumer trigger)
MAX_VALUEfor the size and this
TaskMemoryManageris requested to acquire execution memory (and trySpillAndAcquire)
ShuffleExternalSorteris requested to growPointerArrayIfNecessary, insertRecord
UnsafeExternalSorteris requested to createWithExistingInMemorySorter, growPointerArrayIfNecessary, insertRecord, merge
MemoryConsumer takes the following to be created:
- Page Size
MemoryConsumer is an abstract class and cannot be created directly. It is created indirectly for the concrete MemoryConsumers.