Skip to content


BroadcastManager is a Spark service to manage[]s in a Spark application.

BroadcastManager, SparkEnv and BroadcastFactory

BroadcastManager assigns <> to broadcast variables.

BroadcastManager is used to create a[MapOutputTrackerMaster]

== [[creating-instance]] Creating Instance

BroadcastManager takes the following to be created:

  • <> flag
  • [[conf]][SparkConf]
  • [[securityManager]] SecurityManager

When created, BroadcastManager <>.

BroadcastManager is created when SparkEnv is[created] (for the driver and executors and hence the need for the <> flag).

== [[isDriver]] isDriver Flag

BroadcastManager is given isDriver flag when <>.

The isDriver flag indicates whether the initialization happens on the driver (true) or executors (false).

BroadcastManager uses the flag when requested to <> for the <> to[initialize].

== [[broadcastFactory]] TorrentBroadcastFactory

BroadcastManager manages a[BroadcastFactory]:

  • It is created and initialized in <>

  • It is stopped in <> (and that is all it does)

BroadcastManager uses the BroadcastFactory when requested to <> and <>.

== [[cachedValues]] cachedValues Registry


cachedValues: ReferenceMap

== [[nextBroadcastId]] Unique Identifiers of Broadcast Variables

BroadcastManager tracks broadcast variables and controls their identifiers.

Every <> is given a new and unique identifier.

== [[initialize]][[initialized]] Initializing BroadcastManager

[source, scala]

initialize(): Unit

initialize creates a <> and requests it to[initialize].

initialize turns initialized internal flag on to guard against multiple initializations. With the initialized flag already enabled, initialize does nothing.

initialize is used once when BroadcastManager is <>.

== [[stop]] Stopping BroadcastManager

[source, scala]

stop(): Unit

stop requests the <> to[stop].

== [[newBroadcast]] Creating Broadcast Variable

[source, scala]

newBroadcastT: Broadcast[T]

newBroadcast requests the[current BroadcastFactory for a new broadcast variable].

The BroadcastFactory is created when <>.

newBroadcast is used when:

  • MapOutputTracker utility is used to[serializeMapStatuses]

  • SparkContext is requested for a[new broadcast variable]

Last update: 2020-11-27
Back to top