MemoryStore is the memory store for blocks of data.
MemoryStore uses Java’s java.util.LinkedHashMap with access-order ordering mode. In access-order, the order of iteration is the order in which the entries were last accessed, from least-recently accessed to most-recently. That gives LRU cache behaviour when evicting blocks.
Java’s java.util.LinkedHashMap of
|FIXME Where are these dependencies used?|
Add the following line to
Refer to Logging.
releaseUnrollMemoryForThisTask(memoryMode: MemoryMode, memory: Long = Long.MaxValue): Unit
putIteratorAsBytes[T]( blockId: BlockId, values: Iterator[T], classTag: ClassTag[T], memoryMode: MemoryMode): Either[PartiallySerializedBlock[T], Long]
putIteratorAsBytes tries to put the
blockId block in memory store as bytes.
putBytes[T]( blockId: BlockId, size: Long, memoryMode: MemoryMode, _bytes: () => ChunkedByteBuffer): Boolean
putBytes first makes sure that
blockId block has not been registered already in entries internal registry.
putBytes then requests
size memory for the
blockId block in a given
memoryMode from the current
_bytes byte buffer and makes sure that the size is exactly
size. It then registers a
SerializedMemoryEntry (for the bytes and
blockId in the internal entries registry.
You should see the following INFO message in the logs:
INFO Block [blockId] stored as bytes in memory (estimated size [size], free [bytes])
true only after
blockId was successfully registered in the internal entries registry.
|Spark Property||Default Value||Description|
Initial per-task memory size needed to store a block in memory.
Max memory [maxMemory] is less than the initial memory threshold [unrollMemoryThreshold] needed to store a block in memory. Please configure Spark with more memory.
evictBlocksToFreeSpace( blockId: Option[BlockId], space: Long, memoryMode: MemoryMode): Long
contains(blockId: BlockId): Boolean
contains is positive (
true) when the entries internal registry contains
putIteratorAsValues[T]( blockId: BlockId, values: Iterator[T], classTag: ClassTag[T]): Either[PartiallyUnrolledIterator[T], Long]
putIteratorAsValues makes sure that the
BlockId does not exist or throws an
requirement failed: Block [blockId] is already present in the MemoryStore
putIteratorAsValues tries to put the
blockId block in memory store as
MemoryStore takes the following when created:
MemoryStore initializes the internal registries and counters.
reserveUnrollMemoryForThisTask( blockId: BlockId, memory: Long, memoryMode: MemoryMode): Boolean