SchedulerBackend¶

SchedulerBackend is an abstraction of task scheduling backends that can revive resource offers from cluster managers.

SchedulerBackend abstraction allows TaskSchedulerImpl to use variety of cluster managers (with their own resource offers and task scheduling modes).

Note

Being a scheduler backend system assumes a Apache Mesos-like scheduling model in which "an application" gets resource offers as machines become available so it is possible to launch tasks on them. Once required resource allocation is obtained, the scheduler backend can start executors.

Contract¶

applicationAttemptId¶

applicationAttemptId(): Option[String]

Execution attempt ID of this Spark application

Default: None (undefined)

Used when:

TaskSchedulerImpl is requested for the execution attempt ID of a Spark application

applicationId¶

applicationId(): String

Unique identifier of this Spark application

Default: spark-application-[currentTimeMillis]

Used when:

TaskSchedulerImpl is requested for the unique identifier of a Spark application

Default Parallelism¶

defaultParallelism(): Int

Default parallelism, i.e. a hint for the number of tasks in stages while sizing jobs

Used when:

TaskSchedulerImpl is requested for the default parallelism

getDriverAttributes¶

getDriverAttributes: Option[Map[String, String]]

Default: None

Used when:

SparkContext is requested to postApplicationStart

getDriverLogUrls¶

getDriverLogUrls: Option[Map[String, String]]

Driver log URLs

Default: None (undefined)

Used when:

SparkContext is requested to postApplicationStart

isReady¶

isReady(): Boolean

Controls whether this SchedulerBackend is ready (true) or not (false)

Default: true

Used when:

TaskSchedulerImpl is requested to wait until scheduling backend is ready

Killing Task¶

killTask(
  taskId: Long,
  executorId: String,
  interruptThread: Boolean,
  reason: String): Unit

Kills a given task

Default: UnsupportedOperationException

Used when:

TaskSchedulerImpl is requested to killTaskAttempt and killAllTaskAttempts
TaskSetManager is requested to handle a successful task attempt

Maximum Number of Concurrent Tasks¶

maxNumConcurrentTasks(
  rp: ResourceProfile): Int

Maximum number of concurrent tasks that can be launched (based on the given ResourceProfile)

See:

Used when:

SparkContext is requested for the maximum number of concurrent tasks

reviveOffers¶

reviveOffers(): Unit

Handles resource allocation offers (from the scheduling system)

Used when TaskSchedulerImpl is requested to:

Starting SchedulerBackend¶

start(): Unit

Starts this SchedulerBackend

Used when:

TaskSchedulerImpl is requested to start

stop¶

stop(): Unit

Stops this SchedulerBackend

Used when:

TaskSchedulerImpl is requested to stop

Implementations¶

CoarseGrainedSchedulerBackend
LocalSchedulerBackend
MesosFineGrainedSchedulerBackend