Skip to content

ExternalClusterManager

ExternalClusterManager is an abstraction of pluggable cluster managers that can create a SchedulerBackend and TaskScheduler for a given master URL (when SparkContext is created).

Note

The support for pluggable cluster managers was introduced in SPARK-13904 Add support for pluggable cluster manager.

ExternalClusterManager can be registered using the java.util.ServiceLoader mechanism (with service markers under META-INF/services directory).

Contract

Checking Support for Master URL

canCreate(
  masterURL: String): Boolean

Checks whether this cluster manager instance can create scheduler components for a given master URL

Used when SparkContext is created (and requested for a cluster manager)

Creating SchedulerBackend

createSchedulerBackend(
  sc: SparkContext,
  masterURL: String,
  scheduler: TaskScheduler): SchedulerBackend

Creates a SchedulerBackend for a given SparkContext, master URL, and TaskScheduler.

Used when SparkContext is created (and requested for a SchedulerBackend and TaskScheduler)

Creating TaskScheduler

createTaskScheduler(
  sc: SparkContext,
  masterURL: String): TaskScheduler

Creates a TaskScheduler for a given SparkContext and master URL

Used when SparkContext is created (and requested for a SchedulerBackend and TaskScheduler)

Initializing Scheduling Components

initialize(
  scheduler: TaskScheduler,
  backend: SchedulerBackend): Unit

Initializes the TaskScheduler and SchedulerBackend

Used when SparkContext is created (and requested for a SchedulerBackend and TaskScheduler)

Implementations