Skip to content

BlockId

BlockId is an abstraction of data block identifiers based on an unique name.

Contract

Name

name: String

A globally unique identifier of this Block

Used when:

Implementations

Sealed Abstract Class

BlockId is a Scala sealed abstract class which means that all of the implementations are in the same compilation unit (a single file).

Learn more in the Scala Language Specification.

BroadcastBlockId

BlockId for broadcast variable blocks:

  • broadcastId identifier
  • Optional field name (default: empty)

Uses broadcast_ prefix for the name

Used when:

CacheId

BlockId for...FIXME

PythonStreamBlockId

BlockId for...FIXME

RDDBlockId

BlockId for RDD partitions:

  • rddId identifier
  • splitIndex identifier

Uses rdd_ prefix for the name

Used when:

Compressed when spark.rdd.compress configuration property is enabled

ShuffleBlockBatchId

BlockId for...FIXME

ShuffleBlockChunkId

BlockId for shuffle block chunks in Push-Based Shuffle:

  • shuffleId identifier
  • shuffleMergeId identifier
  • reduceId identifier
  • chunkId identifier

Uses shuffleChunk_[shuffleId]_[shuffleMergeId]_[reduceId]_[chunkId] pattern for the name

ShuffleBlockId

BlockId for shuffle blocks:

  • shuffleId identifier
  • mapId identifier
  • reduceId identifier

Uses shuffle_ prefix for the name

Used when:

Compressed when spark.shuffle.compress configuration property is enabled

ShuffleChecksumBlockId

BlockId for...FIXME

ShuffleDataBlockId

BlockId for...FIXME

ShuffleIndexBlockId

BlockId for...FIXME

ShuffleMergedBlockId

BlockId for...FIXME

ShuffleMergedDataBlockId

BlockId for...FIXME

ShuffleMergedIndexBlockId

BlockId for...FIXME

ShuffleMergedMetaBlockId

BlockId for...FIXME

ShufflePushBlockId

BlockId for...FIXME

StreamBlockId

BlockId for...FIXME:

  • streamId
  • uniqueId

Uses the following name:

input-[streamId]-[uniqueId]

Used in Spark Streaming

TaskResultBlockId

BlockId for...FIXME

TempLocalBlockId

BlockId for...FIXME

TempShuffleBlockId

BlockId for...FIXME

TestBlockId

BlockId for...FIXME

Creating BlockId by Name

apply(
  name: String): BlockId

apply creates one of the available BlockIds by the given name (that uses a prefix to differentiate between different BlockIds).

apply is used when: