StateStoreAwareZipPartitionsRDD¶
StateStoreAwareZipPartitionsRDD
is a ZippedPartitionsRDD2
with the <
StateStoreAwareZipPartitionsRDD
is <StreamingSymmetricHashJoinExec
physical operator is requested to <
Creating Instance¶
StateStoreAwareZipPartitionsRDD
takes the following to be created:
- [[sc]]
SparkContext
- [[f]] Function (
(Iterator[A], Iterator[B]) => Iterator[V]
, e.g. processPartitions) - [[rdd1]] Left RDD - the RDD of the left side of a join (
RDD[A]
) - [[rdd2]] Right RDD - the RDD of the right side of a join (
RDD[B]
) - [[stateInfo]] StatefulOperatorStateInfo
- [[stateStoreNames]] Names of the state stores
- [[storeCoordinator]] StateStoreCoordinatorRef
=== [[getPreferredLocations]] Placement Preferences of Partition (Preferred Locations) -- getPreferredLocations
Method
[source, scala]¶
getPreferredLocations(partition: Partition): Seq[String]¶
NOTE: getPreferredLocations
is a part of the RDD Contract to specify placement preferences (aka preferred task locations), i.e. where tasks should be executed to be as close to the data as possible.
getPreferredLocations
simply requests the <