CreateNamedStruct¶

CreateNamedStruct is an Expression.

Creating Instance¶

CreateNamedStruct takes the following to be created:

Child Expressions

CreateNamedStruct is created when:

SerializerBuildHelper utility is used to createSerializerForObject
ResolveExpressionsWithNamePlaceholders logical analysis rule is executed
ResolveUnion logical analysis rule is executed
Catalyst DSL's namedStruct operator is used
ExpressionEncoder is created
RowEncoder utility is used to create a serializer for a StructType
CreateStruct utility is used to create a CreateNamedStruct
ObjectSerializerPruning logical optimization is executed
many many others

Never Nullable¶

nullable: Boolean

nullable is always disabled (false).

nullable is part of the Expression abstraction.

Node Patterns¶

nodePatterns: Seq[TreePattern]

nodePatterns is CREATE_NAMED_STRUCT.

nodePatterns is part of the TreeNode abstraction.

Pretty Name¶

prettyName: String

prettyName is either function alias for the functionAliasName tag (if defined) or named_struct.

prettyName is part of the Expression abstraction.

Interpreted Expression Evaluation¶

eval(
  input: InternalRow): Any

eval creates an InternalRow with the result of evaluation of all the value expressions.

eval is part of the Expression abstraction.

Code-Generated Expression Evaluation¶

doGenCode(
  ctx: CodegenContext,
  ev: ExprCode): ExprCode

doGenCode is part of the Expression abstraction.

import org.apache.spark.sql.functions.lit
val exprs = Seq("a", 1).map(lit).map(_.expr)

import org.apache.spark.sql.catalyst.dsl.expressions._
val ns = namedStruct(exprs: _*)

// doGenCode is used when Expression.genCode is executed

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext
val code = ns.genCode(ctx).code

scala> println(code)
Object[] values_0 = new Object[1];


if (false) {
  values_0[0] = null;
} else {
  values_0[0] = 1;
}

final InternalRow value_0 = new org.apache.spark.sql.catalyst.expressions.GenericInternalRow(values_0);
values_0 = null;

FunctionRegistry¶

CreateNamedStruct is registered in FunctionRegistry under the name of named_struct SQL function.

import org.apache.spark.sql.catalyst.FunctionIdentifier
val fid = FunctionIdentifier(funcName = "named_struct")
val className = spark.sessionState.functionRegistry.lookupFunction(fid).get.getClassName

scala> println(className)
org.apache.spark.sql.catalyst.expressions.CreateNamedStruct

val q = sql("SELECT named_struct('id', 0)")
// analyzed so the function is resolved already (using FunctionRegistry)
val analyzedPlan = q.queryExecution.analyzed

scala> println(analyzedPlan.numberedTreeString)
00 Project [named_struct(id, 0) AS named_struct(id, 0)#7]
01 +- OneRowRelation

val e = analyzedPlan.expressions.head.children.head
import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct
assert(e.isInstanceOf[CreateNamedStruct])

Catalyst DSL¶

Catalyst DSL defines namedStruct operator to create a CreateNamedStruct expression.

import org.apache.spark.sql.catalyst.dsl.expressions._
val s = namedStruct()

import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct
assert(s.isInstanceOf[CreateNamedStruct])

val s = namedStruct("*")

scala> println(s)
named_struct(*)