MemoryConsumer

MemoryConsumer is an abstraction of spillable memory consumers of TaskMemoryManager.

MemoryConsumers correspond to individual operators and data structures within a task. The TaskMemoryManager receives memory allocation requests from MemoryConsumers and issues callbacks to consumers in order to trigger spilling when running low on memory.

A MemoryConsumer basically tracks how much memory is allocated.

Creating a MemoryConsumer requires a TaskMemoryManager with optional pageSize and a MemoryMode.

Creating Instance

MemoryConsumer takes the following to be created:

MemoryConsumer initializes the internal properties.

Extensions

Table 1. MemoryConsumers (Direct Implementations and Extensions Only)
MemoryConsumer Description

BytesToBytesMap

Used in Spark SQL

HybridRowQueue

LongToUnsafeRowMap

RowBasedKeyValueBatch

ShuffleExternalSorter

Spillable

UnsafeExternalSorter

Spilling

long spill(
  long size,
  MemoryConsumer trigger)
FIXME

Memory Allocated (Used)

used is the amount of memory in use (i.e. allocated) by the MemoryConsumer.

Deallocate MemoryBlock

void freePage(
  MemoryBlock page)

freePage is a protected method to deallocate the MemoryBlock.

Internally, it decrements used registry by the size of page and frees the page.

Allocating Array

LongArray allocateArray(
  long size)

allocateArray allocates LongArray of size length.

Internally, it allocates a page for the requested size. The size is recorded in the internal used counter.

However, if it was not possible to allocate the size memory, it shows the current memory usage and a OutOfMemoryError is thrown.

Unable to acquire [required] bytes of memory, got [got]

allocateArray is used when:

Acquiring Execution Memory (Allocating Memory)

long acquireMemory(
  long size)

acquireMemory requests the TaskMemoryManager to acquire execution memory (of the given size).

The memory allocated is then added to the used internal registry.

acquireMemory is used when:

  • Spillable is requested to maybeSpill

  • Spark SQL’s LongToUnsafeRowMap is requested to ensureAcquireMemory

Allocating Memory Block (Page)

MemoryBlock allocatePage(
  long required)

allocatePage…​FIXME

allocatePage is used when…​FIXME

Throwing OutOfMemoryError

void throwOom(
  MemoryBlock page,
  long required)

throwOom…​FIXME

throwOom is used when MemoryConsumer is requested to allocate an array or a memory block (page) and failed to acquire enough.