AQEOptimizer — AQE Logical Optimizer¶

AQEOptimizer is the logical optimizer for optimizing logical plans in Adaptive Query Execution.

AQEOptimizer is used in AdaptiveSparkPlanExec physical operator.

AQEOptimizer uses spark.sql.adaptive.optimizer.excludedRules configuration property to exclude logical optimizations from the batches of logical optimization rules.

AQEOptimizer is a Catalyst RuleExecutor of the logical (optimization) rules (RuleExecutor[LogicalPlan]).

Creating Instance¶

AQEOptimizer takes the following to be created:

• SQLConf

AQEOptimizer is created alongside AdaptiveSparkPlanExec physical operator.

Default Batches¶

defaultBatches: Seq[Batch]

AQEOptimizer creates a collection of batches with logical optimizations (in the order of their execution):

1. Propagate Empty Relations
2. Dynamic Join Selection
3. Eliminate Limits
4. Optimize One Row Plan

defaultBatches is used as the batches.

Dynamic Join Selection¶

Dynamic Join Selection is a once-executed batch of the following rules:

• DynamicJoinSelection

Eliminate Limits¶

Eliminate Limits is a fixed-point batch of the following rules:

• EliminateLimits

Optimize One Row Plan¶

Optimize One Row Plan is a fixed-point batch of the following rules:

• OptimizeOneRowPlan

Propagate Empty Relations¶

Propagate Empty Relations is a fixed-point batch of the following rules:

• AQEPropagateEmptyRelation
• ConvertToLocalRelation
• UpdateAttributeNullability

Creating FixedPoint Batch Execution Strategy¶

fixedPoint: FixedPoint

fixedPoint creates a FixedPoint batch execution strategy with the following:

Batches¶

batches: Seq[Batch]

batches is part of the RuleExecutor abstraction.

batches returns the default rules excluding the ones specified in the spark.sql.adaptive.optimizer.excludedRules configuration property.

For excluded rules, batches prints out the following INFO message to the logs:

Optimization rule '[ruleName]' is excluded from the optimizer.

For batches with all rules excluded, batches prints out the following INFO message to the logs:

Optimization batch '[name]' is excluded from the optimizer as all enclosed rules have been excluded.

Logging¶

Enable ALL logging level for org.apache.spark.sql.execution.adaptive.AQEOptimizer logger to see what happens inside.

Add the following line to conf/log4j2.properties:

logger.AQEOptimizer.name = org.apache.spark.sql.execution.adaptive.AQEOptimizer
logger.AQEOptimizer.level = all

Refer to Logging.