BlockId

BlockId is an abstraction of data block identifiers based on an unique name.

BlockId is a Scala sealed abstract class and so all the possible implementations are in the single Scala file alongside BlockId.

Contract

Unique Name

name: String

Used when:

Available BlockIds

BroadcastBlockId

BlockId for Broadcast Variables with broadcastId identifier and optional field name (default: empty)

Uses broadcast_ prefix for the name

Used when:

Compressed when spark.broadcast.compress configuration property is enabled

RDDBlockId

BlockId for RDD partitions with rddId and splitIndex identifiers

Uses rdd_ prefix for the name

Used when:

Compressed when spark.rdd.compress configuration property is enabled (default: false)

ShuffleBlockId

BlockId for FIXME with shuffleId, mapId, and reduceId identifiers

Uses shuffle_ prefix for the name

Used when:

Compressed when spark.shuffle.compress configuration property is enabled (default: true)

ShuffleDataBlockId

ShuffleIndexBlockId

StreamBlockId

TaskResultBlockId

TempLocalBlockId

TempShuffleBlockId

apply Factory Method

apply(
  name: String): BlockId

apply creates one of the available BlockIds by the given name (that uses a prefix to differentiate between different BlockIds).

apply is used when: