StateStoreAwareZipPartitionsRDD¶

StateStoreAwareZipPartitionsRDD is a ZippedPartitionsRDD2 with the <> and <> parent RDDs.

StateStoreAwareZipPartitionsRDD is <> exclusively when StreamingSymmetricHashJoinExec physical operator is requested to <> (and requests <> for one).

Creating Instance¶

StateStoreAwareZipPartitionsRDD takes the following to be created:

• [[sc]] SparkContext
• [[f]] Function ((Iterator[A], Iterator[B]) => Iterator[V], e.g. processPartitions)
• [[rdd1]] Left RDD - the RDD of the left side of a join (RDD[A])
• [[rdd2]] Right RDD - the RDD of the right side of a join (RDD[B])
• [[stateInfo]] StatefulOperatorStateInfo
• [[stateStoreNames]] Names of the state stores
• [[storeCoordinator]] StateStoreCoordinatorRef

=== [[getPreferredLocations]] Placement Preferences of Partition (Preferred Locations) -- getPreferredLocations Method

[source, scala]¶

getPreferredLocations(partition: Partition): Seq[String]¶

NOTE: getPreferredLocations is a part of the RDD Contract to specify placement preferences (aka preferred task locations), i.e. where tasks should be executed to be as close to the data as possible.

getPreferredLocations simply requests the <> for the location of every <> (with the <> and the partition ID) and returns unique executor IDs (so that processing a partition happens on the executor with the proper state store for the operator and the partition).