Skip to content


UnresolvedGenerator is a Generator that represents an unresolved generator in a logical query plan.

UnresolvedGenerator is <> exclusively when AstBuilder is requested to sql/[withGenerate] (as part of[Generate] logical operator) for SQL's LATERAL VIEW (in SELECT or FROM clauses).

[[resolved]] UnresolvedGenerator can never be[resolved] (and is replaced at <>).

[[Unevaluable]][[eval]][[doGenCode]] Given UnresolvedGenerator can never be resolved it should not come as a surprise that it[cannot be evaluated] either (i.e. produce a value given an internal row). When requested to evaluate, UnresolvedGenerator simply reports a UnsupportedOperationException.

Cannot evaluate expression: [this]

[[analysis-phase]] [NOTE] ==== UnresolvedGenerator is resolved to a concrete expressions/[Generator] expression when ResolveFunctions logical resolution rule is executed. ====

NOTE: UnresolvedGenerator is similar to[UnresolvedFunction] and differs mostly by the type (to make Spark development with Scala easier?)

=== [[creating-instance]] Creating UnresolvedGenerator Instance

UnresolvedGenerator takes the following when created:

  • [[name]] FunctionIdentifier
  • [[children]] Child[expressions]


// SQLs borrowed from
val sqlText = """
  SELECT pageid, adid
  FROM pageAds LATERAL VIEW explode(adid_list) adTable AS adid
// Register pageAds table
  ("front_page", Array(1, 2, 3)),
  ("contact_page", Array(3, 4, 5)))
  .toDF("pageid", "adid_list")
val query = sql(sqlText)
val logicalPlan = query.queryExecution.logical
scala> println(logicalPlan.numberedTreeString)
00 'Project ['pageid, 'adid]
01 +- 'Generate 'explode('adid_list), false, adtable, ['adid]
02    +- 'UnresolvedRelation `pageAds`

import org.apache.spark.sql.catalyst.plans.logical.Generate
val generator = logicalPlan.collectFirst { case g: Generate => g.generator }.get

scala> :type generator

import org.apache.spark.sql.catalyst.analysis.UnresolvedGenerator
scala> generator.isInstanceOf[UnresolvedGenerator]
res1: Boolean = true