Skip to content

Broadcast

Broadcast[T] is an abstraction of broadcast variables (with the value of type T).

Contract

Destroying Variable

doDestroy(
  blocking: Boolean): Unit

Destroys all the data and metadata related to this broadcast variable

Used when:

  • Broadcast is requested to destroy

Unpersisting Variable

doUnpersist(
  blocking: Boolean): Unit

Deletes the cached copies of this broadcast value on executors

Used when:

Broadcast Value

getValue(): T

Gets the broadcast value

Used when:

  • Broadcast is requested for the value

Implementations

Creating Instance

Broadcast takes the following to be created:

  • Unique Identifier
Abstract Class

Broadcast is an abstract class and cannot be created directly. It is created indirectly for the concrete Broadcasts.

Serializable

Broadcast is a Serializable (Java) so it can be serialized (converted to bytes) and send over the wire from the driver to executors.

Destroying

destroy(): Unit // (1)
destroy(
  blocking: Boolean): Unit
  1. Non-blocking destroy (blocking is false)

destroy removes persisted data and metadata associated with this broadcast variable.

Note

Once a broadcast variable has been destroyed, it cannot be used again.

Unpersisting

unpersist(): Unit // (1)
unpersist(
  blocking: Boolean): Unit
  1. Non-blocking unpersist (blocking is false)

unpersist...FIXME

Brodcast Value

value: T

value makes sure that it was not destroyed and gets the value.

Text Representation

toString: String

toString uses the id as follows:

Broadcast([id])

Validation

Broadcast is considered valid until destroyed.

Broadcast throws a SparkException (with the text representation) when destroyed but requested for the value, to unpersist or destroy:

Attempted to use [toString] after it was destroyed ([destroySite])