PushDownOperatorsToDataSource Logical Optimization¶

PushDownOperatorsToDataSource is a logical optimization that <> (i.e. <>) (before planning so that data source can report statistics more accurately).

Technically, PushDownOperatorsToDataSource is a <> for transforming <>, i.e. Rule[LogicalPlan].

PushDownOperatorsToDataSource is part of the Push down operators to data source scan once-executed rule batch of the SparkOptimizer.

=== [[apply]] Executing Rule -- apply Method

[source, scala]¶

apply(plan: LogicalPlan): LogicalPlan¶

apply...FIXME

apply is part of the Rule abstraction.

=== [[pushDownRequiredColumns]] pushDownRequiredColumns Internal Method

[source, scala]¶

pushDownRequiredColumns(plan: LogicalPlan, requiredByParent: AttributeSet): LogicalPlan¶

pushDownRequiredColumns branches off per the input <> (that is supposed to have at least one child node):

. For <> unary logical operator, pushDownRequiredColumns takes the <> of the <> as the required columns (attributes) and executes itself recursively on the <> + Note that the input requiredByParent attributes are not considered in the required columns.

. For Filter unary logical operator, pushDownRequiredColumns adds the <> of the filter condition to the input requiredByParent attributes and executes itself recursively on the child logical operator

. For <> unary logical operator, pushDownRequiredColumns...FIXME

. For other logical operators, pushDownRequiredColumns simply executes itself (using TreeNode.mapChildren) recursively on the child nodes (logical operators)

pushDownRequiredColumns is used when PushDownOperatorsToDataSource logical optimization is requested to <>.

=== [[FilterAndProject]][[unapply]] Destructuring Logical Operator -- FilterAndProject.unapply Method

[source, scala]¶

unapply(plan: LogicalPlan): Option[(Seq[NamedExpression], Expression, DataSourceV2Relation)]¶

unapply is part of FilterAndProject extractor object to destructure the input <> into a tuple with...FIXME

unapply works with (matches) the following logical operators:

. For a Filter with a <> leaf logical operator, unapply...FIXME

. For a Filter with a <> over a <> leaf logical operator, unapply...FIXME

. For others, unapply returns None (i.e. does nothing / does not match)

NOTE: unapply is used exclusively when PushDownOperatorsToDataSource logical optimization is requested to <>.