CleanupDynamicPruningFilters Logical Optimization¶

CleanupDynamicPruningFilters is a logical optimization for Dynamic Partition Pruning.

CleanupDynamicPruningFilters is a Rule[LogicalPlan] (a Catalyst Rule for logical operators).

CleanupDynamicPruningFilters is part of the Cleanup filters that cannot be pushed down batch of the SparkOptimizer.

Executing Rule¶

apply(
  plan: LogicalPlan): LogicalPlan

apply is part of the Rule abstraction.

spark.sql.optimizer.dynamicPartitionPruning.enabled

apply is a noop (does nothing and returns the given LogicalPlan) when executed with spark.sql.optimizer.dynamicPartitionPruning.enabled configuration property disabled.

apply finds logical operators with the following tree patterns:

apply transforms the given logical plan as follows:

For LogicalRelation logical operators over HadoopFsRelations, apply removeUnnecessaryDynamicPruningSubquery
For HiveTableRelation logical operators, apply removeUnnecessaryDynamicPruningSubquery
For DataSourceV2ScanRelation logical operators, apply removeUnnecessaryDynamicPruningSubquery
DynamicPruning predicate expressions in Filter logical operators are replaced with true literals (cleaned up)