BroadcastManager

Broadcast Manager (BroadcastManager) is a Spark service to manage broadcast variables in Spark. It is created for a Spark application when SparkContext is initialized and is a simple wrapper around BroadcastFactory.

BroadcastManager tracks the number of broadcast variables in a Spark application (using the internal field nextBroadcastId).

The idea is to transfer values used in transformations from a driver to executors in a most effective way so they are copied once and used many times by tasks (rather than being copied every time a task is launched).

When initialized, BroadcastManager creates an instance of TorrentBroadcastFactory.

stop Method

FIXME

Creating BroadcastManager Instance

FIXME

Initializing BroadcastManager — initialize Internal Method

initialize(): Unit
initialize is executed only once (when BroadcastManager is created) and controlled by the internal initialized flag.

Creating Broadcast Variable

newBroadcast[T](value_ : T, isLocal: Boolean): Broadcast[T]

newBroadcast simply requests the current BroadcastFactory for a new broadcast variable.

The BroadcastFactory is created when BroadcastManager is initialized.

Settings

Table 1. Settings
Name Default value Description

spark.broadcast.blockSize

4m

The size of a block (in kB when unit not specified).

Used when TorrentBroadcast stores brodcast blocks to BlockManager.

spark.broadcast.compress

true

The flag to enable compression.

Refer to CompressionCodec.

Used when TorrentBroadcast is created and later when it stores broadcast blocks to BlockManager. Also in SerializerManager.