CoalesceShufflePartitions Adaptive Physical Optimization¶
CoalesceShufflePartitions can be turned on and off using spark.sql.adaptive.coalescePartitions.enabled configuration property.
CoalesceShufflePartitions takes the following to be created:
CoalesceShufflePartitions is created when:
AdaptiveSparkPlanExecphysical operator is requested for the QueryStage Optimizer Rules
apply( plan: SparkPlan): SparkPlan
spark.sql.adaptive.coalescePartitions.enabled Configuration Property
apply makes sure that one of the following holds or does nothing (and simply gives the input SparkPlan back unmodified):
- All the leaves in the query plan are QueryStageExec physical operators
CustomShuffleReaderExecphysical operator in the given query plan
apply makes sure that the following holds or does nothing (and simply gives the input SparkPlan back unmodified):
- All ShuffleQueryStageExec leaf physical operators use ShuffleExchangeExec unary physical operators that have canChangeNumPartitions flag enabled
apply is part of the Rule abstraction.
updateShuffleReads( plan: SparkPlan, specsMap: Map[Int, Seq[ShufflePartitionSpec]]): SparkPlan
updateShuffleReads is used when: