InjectRuntimeFilter Logical Optimization¶

InjectRuntimeFilter is a logical optimization (i.e., a Rule of LogicalPlan).

InjectRuntimeFilter is part of InjectRuntimeFilter fixed-point batch of rules.

Executing Rule¶

apply(
  plan: LogicalPlan): LogicalPlan

apply is part of the Rule abstraction.

apply tryInjectRuntimeFilter unless one of the following holds (and the rule is a noop):

• The given query plan is a correlated Subquery
• spark.sql.optimizer.runtimeFilter.semiJoinReduction.enabled and spark.sql.optimizer.runtime.bloomFilter.enabled are both disabled

tryInjectRuntimeFilter¶

tryInjectRuntimeFilter(
  plan: LogicalPlan): LogicalPlan

tryInjectRuntimeFilter finds equi-joins in the given LogicalPlan.

When some requirements are met, tryInjectRuntimeFilter injectFilter on the left side first and on the right side if on the left was not successful.

tryInjectRuntimeFilter uses spark.sql.optimizer.runtimeFilter.number.threshold configuration property.

Injecting Filter Operator¶

injectFilter(
  filterApplicationSideExp: Expression,
  filterApplicationSidePlan: LogicalPlan,
  filterCreationSideExp: Expression,
  filterCreationSidePlan: LogicalPlan): LogicalPlan

injectFilter...FIXME

Injecting BloomFilter¶

injectBloomFilter(
  filterApplicationSideExp: Expression,
  filterApplicationSidePlan: LogicalPlan,
  filterCreationSideExp: Expression,
  filterCreationSidePlan: LogicalPlan): LogicalPlan

injectBloomFilter...FIXME

injectInSubqueryFilter¶

injectInSubqueryFilter(
  filterApplicationSideExp: Expression,
  filterApplicationSidePlan: LogicalPlan,
  filterCreationSideExp: Expression,
  filterCreationSidePlan: LogicalPlan): LogicalPlan

injectInSubqueryFilter...FIXME