Skip to content

SchedulerBackend

SchedulerBackend is an abstraction of task scheduling backends that can revive resource offers from cluster managers.

SchedulerBackend abstraction allows TaskSchedulerImpl to use variety of cluster managers (with their own resource offers and task scheduling modes).

Note

Being a scheduler backend system assumes a Apache Mesos-like scheduling model in which "an application" gets resource offers as machines become available so it is possible to launch tasks on them. Once required resource allocation is obtained, the scheduler backend can start executors.

Contract

applicationAttemptId

applicationAttemptId(): Option[String]

Execution attempt ID of this Spark application

Default: None (undefined)

Used when:

applicationId

applicationId(): String

Unique identifier of this Spark application

Default: spark-application-[currentTimeMillis]

Used when:

Default Parallelism

defaultParallelism(): Int

Default parallelism, i.e. a hint for the number of tasks in stages while sizing jobs

Used when:

getDriverAttributes

getDriverAttributes: Option[Map[String, String]]

Default: None

Used when:

getDriverLogUrls

getDriverLogUrls: Option[Map[String, String]]

Driver log URLs

Default: None (undefined)

Used when:

isReady

isReady(): Boolean

Controls whether this SchedulerBackend is ready (true) or not (false)

Default: true

Used when:

Killing Task

killTask(
  taskId: Long,
  executorId: String,
  interruptThread: Boolean,
  reason: String): Unit

Kills a given task

Default: UnsupportedOperationException

Used when:

maxNumConcurrentTasks

maxNumConcurrentTasks(): Int

Maximum number of concurrent tasks that can currently be launched

Used when:

reviveOffers

reviveOffers(): Unit

Handles resource allocation offers (from the scheduling system)

Used when TaskSchedulerImpl is requested to:

Starting SchedulerBackend

start(): Unit

Starts this SchedulerBackend

Used when:

  • TaskSchedulerImpl is requested to start

stop

stop(): Unit

Stops this SchedulerBackend

Used when:

  • TaskSchedulerImpl is requested to stop

Implementations


Last update: 2020-12-13