Skip to content

== [[SchedulingMode]] Scheduling Mode -- spark.scheduler.mode Spark Property

Scheduling Mode (aka order task policy or scheduling policy or scheduling order) defines a policy to sort tasks in order for execution.

The scheduling mode schedulingMode attribute is part of the[TaskScheduler Contract].

The only implementation of the TaskScheduler contract in Spark --[TaskSchedulerImpl] -- uses[spark.scheduler.mode] setting to configure schedulingMode that is merely used to set up the[rootPool] attribute (with FIFO being the default). It happens when[TaskSchedulerImpl is initialized].

There are three acceptable scheduling modes:

  • [[FIFO]] FIFO with no pools but a single top-level unnamed pool with elements being[TaskSetManager] objects; lower priority gets[Schedulable] sooner or earlier stage wins.
  • [[FAIR]] FAIR with a[hierarchy of Schedulable (sub)pools] with the[rootPool] at the top.
  • [[NONE]] NONE (not used)

NOTE: Out of three possible SchedulingMode policies only FIFO and FAIR modes are supported by[TaskSchedulerImpl].


After the root pool is initialized, the scheduling mode is no longer relevant (since the[Schedulable] that represents the root pool is fully set up).

The root pool is later used when[TaskSchedulerImpl submits tasks (as TaskSets) for execution].

NOTE: The[root pool] is a Schedulable. Refer to[Schedulable].

=== [[fair-scheduling-sparkui]] Monitoring FAIR Scheduling Mode using Spark UI

CAUTION: FIXME Describe me...

Last update: 2020-10-06