BlockInfo — Metadata of Memory Block

BlockInfo is a metadata of memory block (aka memory page) — the memory block’s size, the number of readers and the id of the writer task.

BlockInfo has a StorageLevel, ClassTag and tellMaster flag.

Size — size Attribute

size attribute is the size of the memory block. It starts with 0.

It represents the number of bytes that BlockManager saved or BlockManager.doPutIterator.

Reader Count — readerCount Counter

readerCount counter is the number of readers of the memory block, i.e. the number of read locks. It starts with 0.

readerCount is incremented when a read lock is acquired and decreases when the following happens:

Writer Task — writerTask Attribute

writerTask attribute is the task that owns the write lock for the memory block.

A writer task can be one of the three possible identifiers:

  • NO_WRITER (i.e. -1) to denote no writers and hence no write lock in use.

  • NON_TASK_WRITER (i.e. -1024) for non-task threads, e.g. by a driver thread or by unit test code.

  • the task attempt id of the task which currently holds the write lock for this block.

The writer task is assigned in the following scenarios: