Skip to content

ExtractWindowExpressions Logical Resolution Rule

ExtractWindowExpressions is a logical resolution rule that <> and replaces (extracts) <> expressions with <> logical operators.

ExtractWindowExpressions is part of the Resolution fixed-point batch in the standard batches of the Analyzer.

ExtractWindowExpressions is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].

import spark.sessionState.analyzer.ExtractWindowExpressions

// Example 1: Filter + Aggregate with WindowExpressions in aggregateExprs
val q = ???
val plan = q.queryExecution.logical
val afterExtractWindowExpressions = ExtractWindowExpressions(plan)

// Example 2: Aggregate with WindowExpressions in aggregateExprs
val q = ???
val plan = q.queryExecution.logical
val afterExtractWindowExpressions = ExtractWindowExpressions(plan)

// Example 3: Project with WindowExpressions in projectList
val q = ???
val plan = q.queryExecution.logical
val afterExtractWindowExpressions = ExtractWindowExpressions(plan)

=== [[apply]] Executing Rule

[source, scala]

apply(plan: LogicalPlan): LogicalPlan

apply transforms the logical operators downwards in the input <> as follows:

  • For Filter unary operators with Aggregate operator that <> in the <>, apply...FIXME

  • For <> logical operators that <> in the <>, apply...FIXME

  • For <> logical operators that <> in the <>, apply...FIXME

apply is part of the Rule abstraction.

=== [[hasWindowFunction]] hasWindowFunction Internal Method

[source, scala]

hasWindowFunction(projectList: Seq[NamedExpression]): Boolean // <1> hasWindowFunction(expr: NamedExpression): Boolean

<1> Executes the other hasWindowFunction on every NamedExpression in the projectList

hasWindowFunction is positive (true) when the input expr <> is a <> expression. Otherwise, hasWindowFunction is negative (false).

NOTE: hasWindowFunction is used when ExtractWindowExpressions logical resolution rule is requested to <> and <>.

=== [[extract]] extract Internal Method

[source, scala]

extract(expressions: Seq[NamedExpression]): (Seq[NamedExpression], Seq[NamedExpression])

extract...FIXME

NOTE: extract is used exclusively when ExtractWindowExpressions logical resolution rule is <>.

=== [[addWindow]] Adding Project and Window Logical Operators to Logical Plan -- addWindow Internal Method

[source, scala]

addWindow( expressionsWithWindowFunctions: Seq[NamedExpression], child: LogicalPlan): LogicalPlan

addWindow adds a <> logical operator with one or more <> logical operators (for every <> in the input <>) to the input <>.

Internally, addWindow...FIXME

NOTE: addWindow is used exclusively when ExtractWindowExpressions logical resolution rule is <>.