|The support for pluggable cluster managers was introduced in SPARK-13904 Add support for pluggable cluster manager.|
External cluster managers are registered using the
java.util.ServiceLoader mechanism (with service markers under
META-INF/services directory). This allows auto-loading implementations of ExternalClusterManager interface.
ExternalClusterManager is a
canCreate(masterURL: String): Boolean
canCreate is a mechanism to match a ExternalClusterManager implementation to a given master URL.
createTaskScheduler(sc: SparkContext, masterURL: String): TaskScheduler
createSchedulerBackend(sc: SparkContext, masterURL: String, scheduler: TaskScheduler): SchedulerBackend
initialize(scheduler: TaskScheduler, backend: SchedulerBackend): Unit
|There is a cyclic dependency between a task scheduler and a backend scheduler that begs for this additional initialization step.|