Skip to content


= [[MemoryConsumer]] MemoryConsumer

MemoryConsumer is an abstraction of <> of[TaskMemoryManager].

MemoryConsumers correspond to individual operators and data structures within a task. The TaskMemoryManager receives memory allocation requests from MemoryConsumers and issues callbacks to consumers in order to trigger <> when running low on memory.

A MemoryConsumer basically tracks <>.

Creating a MemoryConsumer requires a[TaskMemoryManager] with optional pageSize and a MemoryMode.

== [[creating-instance]] Creating Instance

MemoryConsumer takes the following to be created:

  • [[taskMemoryManager]][TaskMemoryManager]
  • [[pageSize]] Page size (in bytes)
  • [[mode]] MemoryMode

MemoryConsumer initializes the <>.

== [[extensions]] Extensions

.MemoryConsumers (Direct Implementations and Extensions Only) [cols="30,70",options="header",width="100%"] |=== | MemoryConsumer | Description

|[BytesToBytesMap] | [[BytesToBytesMap]] Used in Spark SQL

| HybridRowQueue | [[HybridRowQueue]]

| LongToUnsafeRowMap | [[LongToUnsafeRowMap]]

| RowBasedKeyValueBatch | [[RowBasedKeyValueBatch]]

| ShuffleExternalSorter | [[ShuffleExternalSorter]]

|[Spillable] | [[Spillable]]

| UnsafeExternalSorter | [[UnsafeExternalSorter]]


== [[contract]][[spill]] Spilling

[source, java]

long spill( long size, MemoryConsumer trigger)


NOTE: spill is used when[TaskMemoryManager forces MemoryConsumers to release memory when requested to acquire execution memory]

== [[used]][[getUsed]] Memory Allocated (Used)

used is the amount of memory in use (i.e. allocated) by the MemoryConsumer.

== [[freePage]] Deallocate MemoryBlock

[source, java]

void freePage( MemoryBlock page)

freePage is a protected method to deallocate the MemoryBlock.

Internally, it decrements <> registry by the size of page and[frees the page].

== [[allocateArray]] Allocating Array

[source, java]

LongArray allocateArray( long size)

allocateArray allocates LongArray of size length.

Internally, it[allocates a page] for the requested size. The size is recorded in the internal <> counter.

However, if it was not possible to allocate the size memory, it[shows the current memory usage] and a OutOfMemoryError is thrown.

Unable to acquire [required] bytes of memory, got [got]

allocateArray is used when:

  • BytesToBytesMap is requested to[allocate]

  • ShuffleExternalSorter is requested to[growPointerArrayIfNecessary]

  • ShuffleInMemorySorter is[created] and[reset]

  • UnsafeExternalSorter is requested to[growPointerArrayIfNecessary]

  • UnsafeInMemorySorter is[created] and[reset]

  • Spark SQL's UnsafeKVExternalSorter is created

== [[acquireMemory]] Acquiring Execution Memory (Allocating Memory)

[source, java]

long acquireMemory( long size)

acquireMemory requests the <> to[acquire execution memory] (of the given size).

The memory allocated is then added to the <> internal registry.

acquireMemory is used when:

  • Spillable is requested to[maybeSpill]

  • Spark SQL's LongToUnsafeRowMap is requested to ensureAcquireMemory

== [[allocatePage]] Allocating Memory Block (Page)

[source, java]

MemoryBlock allocatePage( long required)


allocatePage is used when...FIXME

== [[throwOom]] Throwing OutOfMemoryError

[source, java]

void throwOom( MemoryBlock page, long required)


throwOom is used when MemoryConsumer is requested to <> or a <> and failed to acquire enough.

Last update: 2020-10-06
Back to top