Skip to content

AQEOptimizer — Adaptive Logical Optimizer

AQEOptimizer is a logical optimizer (a RuleExecutor of the logical rules) for re-optimizing logical plans in Adaptive Query Execution.

Creating Instance

AQEOptimizer takes the following to be created:

AQEOptimizer is created when:

  • AdaptiveSparkPlanExec physical operator is created

Default Batches

Propagate Empty Relations

Dynamic Join Selection

Batches

batches: Seq[Batch]

batches is part of the RuleExecutor abstraction.

batches uses the spark.sql.adaptive.optimizer.excludedRules configuration property for the rules to exclude from the default rules.

For excluded rules, batches prints out the following INFO message to the logs:

Optimization rule '[ruleName]' is excluded from the optimizer.

For batches with all rules excluded, batches prints out the following INFO message to the logs:

Optimization batch '[name]' is excluded from the optimizer as all enclosed rules have been excluded.

Logging

Enable ALL logging level for org.apache.spark.sql.execution.adaptive.AQEOptimizer logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.execution.adaptive.AQEOptimizer=ALL

Refer to Logging.

Back to top