Skip to content

PluginContainer

PluginContainer is an abstraction of plugin containers that can registerMetrics (for the driver and executors).

PluginContainer is created for the driver and executors using apply utility.

Contract

registerMetrics

registerMetrics(
  appId: String): Unit

Used when SparkContext is created

shutdown

shutdown(): Unit

Used when:

  • SparkContext is requested to stop
  • Executor is requested to stop

Implementations

Sealed Abstract Class

PluginContainer is a Scala sealed abstract class which means that all of the implementations are in the same compilation unit (a single file).

Creating PluginContainer

// the driver
apply(
  sc: SparkContext,
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]
// executors
apply(
  env: SparkEnv,
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]
// private helper
apply(
  ctx: Either[SparkContext, SparkEnv],
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]

apply creates a PluginContainer for the driver or executors (based on the type of the first input argument, i.e. SparkContext or SparkEnv, respectively).

apply first loads the SparkPlugins defined by spark.plugins configuration property.

Only when there was at least one plugin loaded, apply creates a DriverPluginContainer or ExecutorPluginContainer.

apply is used when:


Last update: 2020-10-08