Skip to content

EqualNullSafe Predicate Expression

EqualNullSafe is a BinaryComparison predicate expression that represents the following high-level operators in a logical plan:

  • <=> SQL operator
  • Column.<=> operator

Null Safeness

EqualNullSafe is safe for null values, i.e. is like EqualTo with the following "safeness":

  • true if both operands are null
  • false if either operand is null

Creating Instance

EqualNullSafe takes the following to be created:

EqualNullSafe is created when:

Interpreted Expression Evaluation

Expression 
eval(
  input: InternalRow): Any

eval is part of the Expression abstraction.

eval requests the left and right expressions to evaluate with the given InternalRow.

For all the two expressions evaluated to nulls, eval returns true.

For either expression evaluated to null, eval returns false.

Otherwise, eval uses ordering on the data type of the left expression.

Generating Java Source Code for Code-Generated Expression Evaluation

Expression 
doGenCode(
  ctx: CodegenContext,
  ev: ExprCode): ExprCode

doGenCode is part of the Expression abstraction.

doGenCode requests the left and right expressions to generate a source code for expression evaluation.

doGenCode requests the given CodegenContext to generate code for equal expression.

In the end, doGenCode updates the given ExprCode to use the code to evaluate the expression.

import org.apache.spark.sql.catalyst.dsl.expressions._
val right = 'right
val left = 'left
val c = right <=> left

import org.apache.spark.sql.catalyst.expressions.EqualNullSafe
val e = c.expr.asInstanceOf[EqualNullSafe]

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext

// FIXME
scala> e.genCode(ctx)
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object
  at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:227)
  at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:201)
  at scala.Option.getOrElse(Option.scala:201)
  at org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:196)
  at org.apache.spark.sql.catalyst.expressions.EqualNullSafe.doGenCode(predicates.scala:1115)
  at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:201)
  at scala.Option.getOrElse(Option.scala:201)
  at org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:196)
  ... 42 elided

Symbol

BinaryOperator 
symbol: String

symbol is part of the BinaryOperator abstraction.

symbol is <=>.

nullable

Expression 
nullable: Boolean

nullable is part of the Expression abstraction.

nullable is always false.

Catalyst DSL

Catalyst DSL defines <=> operator to create an EqualNullSafe expression.

import org.apache.spark.sql.catalyst.dsl.expressions._
val right = 'right
val left = 'left
val e = right <=> left

scala> e.explain(extended = true)
('right <=> 'left)

scala> println(e.expr.sql)
(`right` <=> `left`)

import org.apache.spark.sql.catalyst.expressions.EqualNullSafe
assert(e.expr.isInstanceOf[EqualNullSafe])