ShuffleSpec¶

ShuffleSpec is an abstraction of shuffle specifications for the following physical optimizations:

• EnsureRequirements
• OptimizeSkewedJoin
• AdaptiveSparkPlanExec

Contract¶

canCreatePartitioning¶

canCreatePartitioning: Boolean

Used when:

• EnsureRequirements physical optimization is executed

isCompatibleWith¶

isCompatibleWith(
  other: ShuffleSpec): Boolean

Used when:

• EnsureRequirements physical optimization is executed
• ValidateRequirements is requested to validate (for OptimizeSkewedJoin physical optimization and AdaptiveSparkPlanExec physical operator)

numPartitions¶

numPartitions: Int

Used when:

• EnsureRequirements physical optimization is executed

Implementations¶

• HashShuffleSpec
• KeyGroupedShuffleSpec
• RangeShuffleSpec
• ShuffleSpecCollection
• SinglePartitionShuffleSpec