WindowExpression¶

WindowExpression is an unevaluable expression that represents a window function over some WindowSpecDefinition.

WindowExpression is created when:

• WindowSpec is requested to withAggregate (when Column.over operator is used)

• WindowsSubstitution logical evaluation rule is executed (with WithWindowDefinition logical operators with UnresolvedWindowExpression expressions)

• AstBuilder is requested to parse a function call or visitPercentile in a SQL statement

WindowExpression can only be with AggregateExpression, AggregateWindowFunction or OffsetWindowFunction expressions which is enforced at analysis.

// Using Catalyst DSL
val wf = 'count.function(star())
val windowSpec = ???

WindowExpression is resolved in ExtractWindowExpressions, ResolveWindowFrame and ResolveWindowOrder logical rules.

WindowExpression is subject to NullPropagation and DecimalAggregates logical optimizations.

Distinct window functions are not supported which is enforced at analysis.

An offset window function can only be evaluated in an ordered row-based window frame with a single offset which is enforced at analysis.

Catalyst DSL¶

windowExpr(
  windowFunc: Expression,
  windowSpec: WindowSpecDefinition): WindowExpression

windowExpr operator in Catalyst DSL creates a WindowExpression expression, e.g. for testing or Spark SQL internals exploration.

Creating Instance¶

WindowExpression takes the following when created:

• Window function expression
• WindowSpecDefinition expression

Demo¶

import org.apache.spark.sql.catalyst.expressions.WindowExpression
// relation - Dataset as a table to query
val table = spark.emptyDataset[Int]

val windowExpr = table
  .selectExpr("count() OVER (PARTITION BY value) AS count")
  .queryExecution
  .logical  // (1)!
  .expressions
  .toList(0)
  .children(0)
  .asInstanceOf[WindowExpression]

1. Use sqlParser directly as in WithWindowDefinition Example

scala> windowExpr.sql
res2: String = count() OVER (PARTITION BY `value` UnspecifiedFrame)