MemoryManager splits available memory into two regions:
Execution memory for computations in shuffles, joins, sorts and aggregations
Storage memory for caching and propagating internal data across Spark nodes
MemoryManager is available as SparkEnv.memoryManager.
import org.apache.spark.SparkEnv val mm = SparkEnv.get.memoryManager
acquireExecutionMemory( numBytes: Long, taskAttemptId: Long, memoryMode: MemoryMode): Long
acquireExecutionMemory tries to acquire up to
numBytes of execution memory for the current task (by
taskAttemptId) and return the number of bytes obtained, or 0 if none can be allocated.
acquireExecutionMemory is used when TaskMemoryManager is requested to acquire execution memory.
acquireStorageMemory( blockId: BlockId, numBytes: Long, memoryMode: MemoryMode): Boolean
acquireStorageMemory tries to acquire
numBytes bytes of memory to cache the given block, evicting existing ones if necessary.
acquireStorageMemory is used when:
acquireUnrollMemory( blockId: BlockId, numBytes: Long, memoryMode: MemoryMode): Boolean
acquireUnrollMemory tries to acquire
numBytes bytes of memory to unroll the given block, evicting existing ones if necessary.
acquireUnrollMemory is used when MemoryStore is requested to reserveUnrollMemoryForThisTask.
maxOffHeapStorageMemory is the total available off-heap memory for storage (in bytes).
maxOffHeapStorageMemory may vary over time.
maxOffHeapStorageMemory is used when:
maxOnHeapStorageMemory is the total available on-heap memory for storage (in bytes).
maxOnHeapStorageMemory may vary over time.
maxOnHeapStorageMemory is used when:
releaseAllExecutionMemoryForTask(taskAttemptId: Long): Long
OFF_HEAP only when the following are all met:
spark.memory.offHeap.enabled configuration property is enabled (it is not by default)
spark.memory.offHeap.size configuration property is greater than
JVM supports unaligned memory access (aka unaligned Unsafe, i.e.
sun.misc.Unsafepackage is available and the underlying system has unaligned-access capability)
Given that spark.memory.offHeap.enabled configuration property is disabled (