Skip to content

PlanDynamicPruningFilters Physical Optimization

PlanDynamicPruningFilters is a physical optimization and part of the default physical optimizations (preparations).

PlanDynamicPruningFilters transforms DynamicPruningSubquery expressions into DynamicPruningExpression expressions.

PlanDynamicPruningFilters is a noop when spark.sql.optimizer.dynamicPartitionPruning.enabled configuration property is disabled (false).

PlanDynamicPruningFilters is a Catalyst Rule for transforming physical operators (Rule[SparkPlan]).

Creating Instance

PlanDynamicPruningFilters takes the following to be created:

PlanDynamicPruningFilters is created when QueryExecution utility is requested for physical query optimizations.

Executing Rule

apply(
  plan: SparkPlan): SparkPlan

apply transforms a DynamicPruningSubquery expression as follows:

  1. Uses QueryExecution.createSparkPlan utility to plan the Build Logical Query (of the DynamicPruningSubquery)

  2. Finds whether to reuse an Exchange (based on spark.sql.exchange.reuse configuration property and other checks)

  3. Creates a DynamicPruningExpression expression

apply is part of the Rule abstraction.