BarrierJobSlotsNumberCheckFailed¶

BarrierJobSlotsNumberCheckFailed is a BarrierJobAllocationFailed with the following exception message:

[SPARK-24819]: Barrier execution mode does not allow run a barrier stage that requires more slots than the total number of slots in the cluster currently.
Please init a new cluster with more resources(e.g. CPU, GPU) or repartition the input RDD(s) to reduce the number of slots required to run this barrier stage.

BarrierJobSlotsNumberCheckFailed can be thrown when DAGScheduler is requested to handle a JobSubmitted event.

Creating Instance¶

BarrierJobSlotsNumberCheckFailed takes the following to be created:

• Required Concurrent Tasks (based on the number of partitions of a barrier RDD)
• Maximum Number of Concurrent Tasks (based on a ResourceProfile used)

BarrierJobSlotsNumberCheckFailed is created when:

• SparkCoreErrors is requested to numPartitionsGreaterThanMaxNumConcurrentTasksError