MemoryStore is the memory store for blocks of data for BlockManager.
releaseUnrollMemoryForThisTask( memoryMode: MemoryMode, memory: Long = Long.MaxValue): Unit
putIteratorAsBytes[T]( blockId: BlockId, values: Iterator[T], classTag: ClassTag[T], memoryMode: MemoryMode): Either[PartiallySerializedBlock[T], Long]
putIteratorAsBytes tries to put the
blockId block in memory store as bytes.
putBytes[T]( blockId: BlockId, size: Long, memoryMode: MemoryMode, _bytes: () => ChunkedByteBuffer): Boolean
putBytes first makes sure that
blockId block has not been registered already in entries internal registry.
putBytes then requests
size memory for the
blockId block in a given
memoryMode from the current
_bytes byte buffer and makes sure that the size is exactly
size. It then registers a
SerializedMemoryEntry (for the bytes and
blockId in the internal entries registry.
You should see the following INFO message in the logs:
INFO Block [blockId] stored as bytes in memory (estimated size [size], free [bytes])
true only after
blockId was successfully registered in the internal entries registry.
contains(blockId: BlockId): Boolean
contains is positive (
true) when the entries internal registry contains
putIteratorAsValues[T]( blockId: BlockId, values: Iterator[T], classTag: ClassTag[T]): Either[PartiallyUnrolledIterator[T], Long]
putIteratorAsValues makes sure that the
BlockId does not exist or throws an
requirement failed: Block [blockId] is already present in the MemoryStore
putIteratorAsValues tries to put the
blockId block in memory store as
reserveUnrollMemoryForThisTask( blockId: BlockId, memory: Long, memoryMode: MemoryMode): Boolean
ALL logging level for
org.apache.spark.storage.memory.MemoryStore logger to see what happens inside.
Add the following line to
Refer to Logging.
entries: LinkedHashMap[BlockId, MemoryEntry[_]]
entries uses access-order ordering mode where the order of iteration is the order in which the entries were last accessed (from least-recently accessed to most-recently). That gives LRU cache behaviour when evicting blocks.