Skip to content

OptimizeSkewInRebalancePartitions Adaptive Physical Optimization

OptimizeSkewInRebalancePartitions is a physical optimization in Adaptive Query Execution.

OptimizeSkewInRebalancePartitions can be turned on and off using spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled configuration property.

Executing Rule

Rule 
apply(
  plan: SparkPlan): SparkPlan

apply is part of the Rule abstraction.

apply works with ShuffleQueryStageExec leaf physical operators only (with the ShuffleExchangeLikes that are supported).

apply tryOptimizeSkewedPartitions of the ShuffleQueryStageExec.

tryOptimizeSkewedPartitions

tryOptimizeSkewedPartitions(
  shuffle: ShuffleQueryStageExec): SparkPlan

tryOptimizeSkewedPartitions...FIXME

Supported ShuffleOrigins

AQEShuffleReadRule 
supportedShuffleOrigins: Seq[ShuffleOrigin]

supportedShuffleOrigins is part of the AQEShuffleReadRule abstraction.

supportedShuffleOrigins is a collection of the following ShuffleOrigins: