LogicalQueryStage Leaf Logical Operator¶

LogicalQueryStage is a leaf logical operator for Adaptive Query Execution.

Creating Instance¶

LogicalQueryStage takes the following to be created:

• Logical Plan
• Physical Plan

LogicalQueryStage is created when:

• AdaptiveSparkPlanExec physical operator is executed

Query Optimization¶

LogicalQueryStage is a "target" of the following logical optimizations:

• AQEPropagateEmptyRelation
• DynamicJoinSelection

Query Planning¶

LogicalQueryStage is planned by LogicalQueryStageStrategy execution planning strategy.

Computing Runtime Statistics¶

computeStats(): Statistics

computeStats tries to find the first QueryStageExec leaf physical operators in the physical plan that is then requested for the statistics.

computeStats prints out the following DEBUG messages to the logs based on the availability of the statistics.

Physical stats available as [physicalStats] for plan: [physicalPlan]

Physical stats not available for plan: [physicalPlan]

In the end, computeStats gives the statistics of the physical operator or requests the logical plan for them.

Logging¶

Enable ALL logging level for org.apache.spark.sql.execution.adaptive.LogicalQueryStage logger to see what happens inside.

Add the following line to conf/log4j2.properties:

logger.LogicalQueryStage.name = org.apache.spark.sql.execution.adaptive.LogicalQueryStage
logger.LogicalQueryStage.level = all

Refer to Logging.