LogicalQueryStage Leaf Logical Operator¶
LogicalQueryStage is a leaf logical operator for Adaptive Query Execution.
LogicalQueryStage takes the following to be created:
LogicalQueryStage is created when:
- AdaptiveSparkPlanExec physical operator is executed
LogicalQueryStage is a "target" of the following logical optimizations:
LogicalQueryStage is planned by LogicalQueryStageStrategy execution planning strategy.
Computing Runtime Statistics¶
computeStats is part of the LeafNode abstraction.
computeStats tries to find the first QueryStageExec leaf physical operators in the physical plan that is then requested for the statistics.
computeStats prints out the following DEBUG messages to the logs based on the availability of the statistics.
Physical stats available as [physicalStats] for plan: [physicalPlan]
Physical stats not available for plan: [physicalPlan]
In the end,
computeStats gives the statistics of the physical operator or requests the logical plan for them.
ALL logging level for
org.apache.spark.sql.execution.adaptive.LogicalQueryStage logger to see what happens inside.
Add the following line to
logger.LogicalQueryStage.name = org.apache.spark.sql.execution.adaptive.LogicalQueryStage logger.LogicalQueryStage.level = all
Refer to Logging.