MapPartitionsWithEvaluatorRDD¶

MapPartitionsWithEvaluatorRDD is an RDD.

Creating Instance¶

MapPartitionsWithEvaluatorRDD takes the following to be created:

• Previous RDD
• PartitionEvaluatorFactory

MapPartitionsWithEvaluatorRDD is created when:

• RDD.mapPartitionsWithEvaluator operator is used
• RDDBarrier.mapPartitionsWithEvaluator operator is used

Computing Partition¶

compute(
  split: Partition,
  context: TaskContext): Iterator[U]

compute requests the PartitionEvaluatorFactory to create a PartitionEvaluator.

compute requests the first parent RDD to iterator.

In the end, compute requests the PartitionEvaluator to evaluate the partition.