Skip to content

WindowExpression

WindowExpression is an unevaluable expression that represents a window function (over some WindowSpecDefinition).

WindowExpression is created when:

WindowExpression can only be with AggregateExpression, AggregateWindowFunction or OffsetWindowFunction expressions which is enforced at analysis.

// Using Catalyst DSL
val wf = 'count.function(star())
val windowSpec = ???

WindowExpression is resolved in ExtractWindowExpressions, ResolveWindowFrame and ResolveWindowOrder logical rules.

WindowExpression is subject to NullPropagation and DecimalAggregates logical optimizations.

Distinct window functions are not supported which is enforced at analysis.

An offset window function can only be evaluated in an ordered row-based window frame with a single offset which is enforced at analysis.

Catalyst DSL

windowExpr(
  windowFunc: Expression,
  windowSpec: WindowSpecDefinition): WindowExpression

windowExpr operator in Catalyst DSL creates a WindowExpression expression, e.g. for testing or Spark SQL internals exploration.

Creating Instance

WindowExpression takes the following when created:

Demo

import org.apache.spark.sql.catalyst.expressions.WindowExpression
// relation - Dataset as a table to query
val table = spark.emptyDataset[Int]

scala> val windowExpr = table
  .selectExpr("count() OVER (PARTITION BY value) AS count")
  .queryExecution
  .logical      // (1)!
  .expressions
  .toList(0)
  .children(0)
  .asInstanceOf[WindowExpression]
windowExpr: org.apache.spark.sql.catalyst.expressions.WindowExpression = 'count() windowspecdefinition('value, UnspecifiedFrame)

scala> windowExpr.sql
res2: String = count() OVER (PARTITION BY `value` UnspecifiedFrame)
  1. Use sqlParser directly as in WithWindowDefinition Example
Back to top