Skip to content

AQEOptimizer — AQE Logical Optimizer

AQEOptimizer is the logical optimizer for optimizing logical plans in Adaptive Query Execution.


AQEOptimizer is used in AdaptiveSparkPlanExec physical operator.

AQEOptimizer uses spark.sql.adaptive.optimizer.excludedRules configuration property to exclude logical optimizations from the batches of logical optimization rules.

AQEOptimizer is a Catalyst RuleExecutor of the logical (optimization) rules (RuleExecutor[LogicalPlan]).

Creating Instance

AQEOptimizer takes the following to be created:

AQEOptimizer is created alongside AdaptiveSparkPlanExec physical operator.

Default Batches

defaultBatches: Seq[Batch]

AQEOptimizer creates a collection of batches with logical optimizations (in the order of their execution):

  1. Propagate Empty Relations
  2. Dynamic Join Selection
  3. Eliminate Limits
  4. Optimize One Row Plan

defaultBatches is used as the batches.

Dynamic Join Selection

Dynamic Join Selection is a once-executed batch of the following rules:

Eliminate Limits

Eliminate Limits is a fixed-point batch of the following rules:

  • EliminateLimits

Optimize One Row Plan

Optimize One Row Plan is a fixed-point batch of the following rules:

  • OptimizeOneRowPlan

Propagate Empty Relations

Propagate Empty Relations is a fixed-point batch of the following rules:

Creating FixedPoint Batch Execution Strategy

fixedPoint: FixedPoint

fixedPoint creates a FixedPoint batch execution strategy with the following:

Attribute Value
maxIterations spark.sql.optimizer.maxIterations
maxIterationsSetting spark.sql.optimizer.maxIterations


batches: Seq[Batch]

batches is part of the RuleExecutor abstraction.

batches returns the default rules excluding the ones specified in the spark.sql.adaptive.optimizer.excludedRules configuration property.

For excluded rules, batches prints out the following INFO message to the logs:

Optimization rule '[ruleName]' is excluded from the optimizer.

For batches with all rules excluded, batches prints out the following INFO message to the logs:

Optimization batch '[name]' is excluded from the optimizer as all enclosed rules have been excluded.


Enable ALL logging level for org.apache.spark.sql.execution.adaptive.AQEOptimizer logger to see what happens inside.

Add the following line to conf/ = org.apache.spark.sql.execution.adaptive.AQEOptimizer
logger.AQEOptimizer.level = all

Refer to Logging.