Skip to content

BroadcastManager

BroadcastManager is a Spark service to manage ROOT:Broadcast.md[]s in a Spark application.

BroadcastManager, SparkEnv and BroadcastFactory

BroadcastManager assigns <> to broadcast variables.

BroadcastManager is used to create a scheduler:MapOutputTrackerMaster.md#BroadcastManager[MapOutputTrackerMaster]

== [[creating-instance]] Creating Instance

BroadcastManager takes the following to be created:

  • <> flag
  • [[conf]] ROOT:SparkConf.md[SparkConf]
  • [[securityManager]] SecurityManager

When created, BroadcastManager <>.

BroadcastManager is created when SparkEnv is core:SparkEnv.md[created] (for the driver and executors and hence the need for the <> flag).

== [[isDriver]] isDriver Flag

BroadcastManager is given isDriver flag when <>.

The isDriver flag indicates whether the initialization happens on the driver (true) or executors (false).

BroadcastManager uses the flag when requested to <> for the <> to TorrentBroadcastFactory.md#initialize[initialize].

== [[broadcastFactory]] TorrentBroadcastFactory

BroadcastManager manages a core:BroadcastFactory.md[BroadcastFactory]:

  • It is created and initialized in <>

  • It is stopped in <> (and that is all it does)

BroadcastManager uses the BroadcastFactory when requested to <> and <>.

== [[cachedValues]] cachedValues Registry

[source,scala]

cachedValues: ReferenceMap

== [[nextBroadcastId]] Unique Identifiers of Broadcast Variables

BroadcastManager tracks broadcast variables and controls their identifiers.

Every <> is given a new and unique identifier.

== [[initialize]][[initialized]] Initializing BroadcastManager

[source, scala]

initialize(): Unit

initialize creates a <> and requests it to core:TorrentBroadcastFactory.md#initialize[initialize].

initialize turns initialized internal flag on to guard against multiple initializations. With the initialized flag already enabled, initialize does nothing.

initialize is used once when BroadcastManager is <>.

== [[stop]] Stopping BroadcastManager

[source, scala]

stop(): Unit

stop requests the <> to core:BroadcastFactory.md#stop[stop].

== [[newBroadcast]] Creating Broadcast Variable

[source, scala]

newBroadcastT: Broadcast[T]


newBroadcast requests the core:BroadcastFactory.md[current BroadcastFactory for a new broadcast variable].

The BroadcastFactory is created when <>.

newBroadcast is used when:

  • MapOutputTracker utility is used to scheduler:MapOutputTracker.md#serializeMapStatuses[serializeMapStatuses]

  • SparkContext is requested for a ROOT:SparkContext.md#broadcast[new broadcast variable]


Last update: 2020-10-06