MapPartitionsRDD¶

MapPartitionsRDD[U, T] is a RDD that transforms (maps) input T records into Us using partition function.

MapPartitionsRDD is a RDD that has exactly one-to-one narrow dependency on the parent RDD.

Creating Instance¶

MapPartitionsRDD takes the following to be created:

• Parent RDD (RDD[T])
• Partition Function
• preservesPartitioning flag
• isFromBarrier Flag
• isOrderSensitive flag

MapPartitionsRDD is created when:

• PairRDDFunctions is requested to mapValues and flatMapValues
• RDD is requested to map, flatMap, filter, glom, mapPartitions, mapPartitionsWithIndexInternal, mapPartitionsInternal, mapPartitionsWithIndex
• RDDBarrier is requested to mapPartitions, mapPartitionsWithIndex

Barrier RDD¶

MapPartitionsRDD can be a barrier RDD in Barrier Execution Mode.

isFromBarrier Flag¶

MapPartitionsRDD can be given isFromBarrier flag when created.

isFromBarrier flag is assumed disabled (false) and can only be enabled (true) using RDDBarrier transformations:

• RDDBarrier.mapPartitions
• RDDBarrier.mapPartitionsWithIndex

isBarrier_¶

isBarrier_ : Boolean

isBarrier_ is enabled (true) when either this MapPartitionsRDD is isFromBarrier or any of the parent RDDs is isBarrier. Otherwise, isBarrier_ is disabled (false).