Skip to content

PluginContainer

PluginContainer is an abstraction of plugin containers that can register metrics (for the driver and executors).

PluginContainer is created for the driver and executors using apply utility.

Contract

Listening to Task Failures

onTaskFailed(
  failureReason: TaskFailedReason): Unit

For ExecutorPluginContainer only

Possible TaskFailedReasons:

  • TaskKilledException
  • TaskKilled
  • FetchFailed
  • TaskCommitDenied
  • ExceptionFailure

Used when:

  • TaskRunner is requested to run (and the task has failed)

Listening to Task Start

onTaskStart(): Unit

For ExecutorPluginContainer only

Used when:

  • TaskRunner is requested to run (and the task has just started)

Listening to Task Success

onTaskSucceeded(): Unit

For ExecutorPluginContainer only

Used when:

  • TaskRunner is requested to run (and the task has finished successfully)

Registering Metrics

registerMetrics(
  appId: String): Unit

Registers metrics for the application ID

For DriverPluginContainer only

Used when:

Shutdown

shutdown(): Unit

Used when:

  • SparkContext is requested to stop
  • Executor is requested to stop

Implementations

Sealed Abstract Class

PluginContainer is a Scala sealed abstract class which means that all of the implementations are in the same compilation unit (a single file).

Creating PluginContainer

// the driver
apply(
  sc: SparkContext,
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]
// executors
apply(
  env: SparkEnv,
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]
// private helper
apply(
  ctx: Either[SparkContext, SparkEnv],
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]

apply creates a PluginContainer for the driver or executors (based on the type of the first input argument, i.e. SparkContext or SparkEnv, respectively).

apply first loads the SparkPlugins defined by spark.plugins configuration property.

Only when there was at least one plugin loaded, apply creates a DriverPluginContainer or ExecutorPluginContainer.

apply is used when: