InterruptibleIterator — Iterator With Support For Task Cancellation

InterruptibleIterator is a custom Scala Iterator that supports task cancellation, i.e. stops iteration when a task was interrupted (cancelled).

Quoting the official Scala Iterator documentation:

Iterators are data structures that allow to iterate over a sequence of elements. They have a hasNext method for checking if there is a next element available, and a next method which returns the next element and discards it from the iterator.

InterruptibleIterator is created when:

InterruptibleIterator takes the following when created:

InterruptibleIterator is a Developer API which is a lower-level, unstable API intended for Spark developers that may change or be removed in minor versions of Apache Spark.

hasNext Method

hasNext: Boolean
hasNext is part of Iterator Contract to test whether this iterator can provide another element.

hasNext requests the TaskContext to kill the task if interrupted (that simply throws a TaskKilledException that in turn breaks the task execution).

In the end, hasNext requests the delegate Iterator to hasNext.

next Method

next(): T
next is part of Iterator Contract to produce the next element of this iterator.

next simply requests the delegate Iterator to next.