PlanAdaptiveDynamicPruningFilters Physical Optimization¶

PlanAdaptiveDynamicPruningFilters is a physical optimization in Adaptive Query Execution.

PlanAdaptiveDynamicPruningFilters is a Catalyst Rule for transforming physical plans (Rule[SparkPlan]).

Creating Instance¶

PlanAdaptiveDynamicPruningFilters takes the following to be created:

• Root AdaptiveSparkPlanExec

PlanAdaptiveDynamicPruningFilters is created when:

• AdaptiveSparkPlanExec leaf physical operator is requested for the adaptive optimizations

Executing Rule¶

apply(
  plan: SparkPlan): SparkPlan

apply is part of the Rule abstraction.

apply is disabled (and simply returns the given SparkPlan unchanged) when the spark.sql.optimizer.dynamicPartitionPruning.enabled is disabled.

apply requests the given SparkPlan to transformAllExpressionsWithPruning in QueryPlans with the DYNAMIC_PRUNING_EXPRESSION and IN_SUBQUERY_EXEC tree patterns:

• DynamicPruningExpressions with InSubqueryExec expressions with SubqueryAdaptiveBroadcastExecs with AdaptiveSparkPlanExec leaf physical operators

In the end, apply creates a new DynamicPruningExpression unary expression (with a InSubqueryExec or TrueLiteral).