OptimizeSkewInRebalancePartitions Adaptive Physical Optimization¶

OptimizeSkewInRebalancePartitions is a physical optimization in Adaptive Query Execution.

OptimizeSkewInRebalancePartitions can be turned on and off using spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled configuration property.

Executing Rule¶

apply(
  plan: SparkPlan): SparkPlan

apply works with ShuffleQueryStageExec leaf physical operators only (with the ShuffleExchangeLikes that are supported).

apply tryOptimizeSkewedPartitions of the ShuffleQueryStageExec.

apply is part of the Rule abstraction.

tryOptimizeSkewedPartitions(
  shuffle: ShuffleQueryStageExec): SparkPlan

tryOptimizeSkewedPartitions...FIXME