MemoryConsumers correspond to individual operators and data structures within a task. The TaskMemoryManager receives memory allocation requests from MemoryConsumers and issues callbacks to consumers in order to trigger spilling when running low on memory.
A MemoryConsumer basically tracks how much memory is allocated.
Creating a MemoryConsumer requires a TaskMemoryManager with optional
pageSize and a
long spill( long size, MemoryConsumer trigger)
LongArray allocateArray( long size)
However, if it was not possible to allocate the
size memory, it shows the current memory usage and a
OutOfMemoryError is thrown.
Unable to acquire [required] bytes of memory, got [got]
allocateArray is used when:
BytesToBytesMap is requested to allocate
ShuffleExternalSorter is requested to growPointerArrayIfNecessary
UnsafeExternalSorter is requested to growPointerArrayIfNecessary
Spark SQL’s UnsafeKVExternalSorter is created
long acquireMemory( long size)
The memory allocated is then added to the used internal registry.
acquireMemory is used when:
Spillable is requested to maybeSpill
Spark SQL’s LongToUnsafeRowMap is requested to ensureAcquireMemory
MemoryBlock allocatePage( long required)
allocatePage is used when…FIXME