ExtractJoinWithBuckets Scala Extractor¶

Destructuring BaseJoinExec¶

unapply(
  plan: SparkPlan): Option[(BaseJoinExec, Int, Int)]

unapply makes sure that the given SparkPlan is a BaseJoinExec and applicable.

If so, unapply getBucketSpec for the left and right join child operators.

unapply...FIXME

unapply is used when:

• CoalesceBucketsInJoin physical optimization is executed

isApplicable¶

isApplicable(
  j: BaseJoinExec): Boolean

isApplicable is true when the following all hold:

1. The given BaseJoinExec physical operator is either a SortMergeJoinExec or a ShuffledHashJoinExec

2. The left side of the join has a FileSourceScanExec operator

3. The right side of the join has a FileSourceScanExec operator

4. satisfiesOutputPartitioning on the leftKeys and the outputPartitioning of the left join operator

5. satisfiesOutputPartitioning on the rightKeys and the outputPartitioning of the right join operator

hasScanOperation¶

hasScanOperation(
  plan: SparkPlan): Boolean

hasScanOperation holds true for SparkPlan physical operators that are FileSourceScanExecs (possibly as the children of FilterExecs and ProjectExecs).

satisfiesOutputPartitioning¶

satisfiesOutputPartitioning(
  keys: Seq[Expression],
  partitioning: Partitioning): Boolean

satisfiesOutputPartitioning holds true for HashPartitioning partitionings that match the given join keys (their number and equivalence).

Bucket Spec of FileSourceScanExec Operator¶

getBucketSpec(
  plan: SparkPlan): Option[BucketSpec]

getBucketSpec finds the FileSourceScanExec operator (in the given SparkPlan) with a non-empty bucket spec but an empty optionalNumCoalescedBuckets. When found, getBucketSpec returns the non-empty bucket spec.