MemoryConsumer¶

MemoryConsumer is an abstraction of memory consumers (of TaskMemoryManager) that support spilling.

MemoryConsumers correspond to individual operators and data structures within a task. TaskMemoryManager receives memory allocation requests from MemoryConsumers and issues callbacks to consumers in order to trigger spilling when running low on memory.

A MemoryConsumer basically tracks how much memory is allocated.

Contract¶

Spilling¶

void spill() // (1)
long spill(
  long size,
  MemoryConsumer trigger)

1. Uses MAX_VALUE for the size and this MemoryConsumer

Used when:

• TaskMemoryManager is requested to acquire execution memory (and trySpillAndAcquire)
• ShuffleExternalSorter is requested to growPointerArrayIfNecessary, insertRecord
• UnsafeExternalSorter is requested to createWithExistingInMemorySorter, growPointerArrayIfNecessary, insertRecord, merge

Implementations¶

• BytesToBytesMap
• ShuffleExternalSorter
• Spillable
• UnsafeExternalSorter
• a few others

Creating Instance¶

MemoryConsumer takes the following to be created:

• TaskMemoryManager
• Page Size
• MemoryMode (ON_HEAP or OFF_HEAP)