Unevaluable Expressions¶
Unevaluable
is an extension of the Expression abstraction for unevaluable expression that cannot be evaluated to produce a value (neither in the interpreted nor code-generated mode).
Unevaluable
expressions are expected to be resolved (replaced) to "evaluable" expressions or logical operators at analysis or optimization phases or they fail analysis.
Unevaluable expressions cannot be evaluated (neither in <
Implementations¶
- AggregateExpression
- DeclarativeAggregate
- DynamicPruningSubquery
- Exists
- HashPartitioning
- ListQuery
- RuntimeReplaceable
- SortOrder
- TimeWindow
- UnresolvedAttribute
- UnresolvedFunction
- UnresolvedOrdinal
- UnresolvedStar
- WindowExpression
- WindowSpecDefinition
- others
Demo¶
/**
Example: Analysis failure due to an Unevaluable expression
UnresolvedFunction is an Unevaluable expression
Using Catalyst DSL to create a UnresolvedFunction
*/
import org.apache.spark.sql.catalyst.dsl.expressions._
val f = 'f.function()
import org.apache.spark.sql.catalyst.dsl.plans._
val logicalPlan = table("t1").select(f)
scala> println(logicalPlan.numberedTreeString)
00 'Project [unresolvedalias('f(), None)]
01 +- 'UnresolvedRelation `t1`
scala> spark.sessionState.analyzer.execute(logicalPlan)
org.apache.spark.sql.AnalysisException: Undefined function: 'f'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.;
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$15$$anonfun$applyOrElse$49.apply(Analyzer.scala:1198)
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$15$$anonfun$applyOrElse$49.apply(Analyzer.scala:1198)
at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:48)
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$15.applyOrElse(Analyzer.scala:1197)
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$15.applyOrElse(Analyzer.scala:1195)