GenerateIdentityValues Non-Deterministic Leaf Expression¶

GenerateIdentityValues is a non-deterministic LeafExpression (Spark SQL).

GenerateIdentityValues uses PartitionIdentityValueGenerator to generate the next IDENTITY value.

Creating Instance¶

GenerateIdentityValues takes the following to be created:

• PartitionIdentityValueGenerator

GenerateIdentityValues is created using apply utility.

Create GenerateIdentityValues¶

apply(
  start: Long,
  step: Long,
  highWaterMarkOpt: Option[Long]): GenerateIdentityValues

apply creates a GenerateIdentityValues with a new PartitionIdentityValueGenerator with the given input arguments.

apply is used when:

• IdentityColumn is requested to createIdentityColumnGenerationExpr

initializeInternal¶

initializeInternal(
  partitionIndex: Int): Unit

initializeInternal requests this PartitionIdentityValueGenerator to initialize with the given partitionIndex.

evalInternal¶

evalInternal(
  input: InternalRow): Long

evalInternal requests this PartitionIdentityValueGenerator for the next IDENTITY value.

Nullable¶

nullable: Boolean

nullable is always false.

Generating Java Source Code for Code-Generated Expression Evaluation¶

doGenCode(
  ctx: CodegenContext,
  ev: ExprCode): ExprCode

doGenCode generates a Java source code with this PartitionIdentityValueGenerator to be initialized (for the current partitionIndex) followed by requesting the next IDENTITY value.

import org.apache.spark.sql.delta.GenerateIdentityValues

val expr = GenerateIdentityValues(start = 0, step = 1, highWaterMarkOpt = None)

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext

val code = expr.genCode(ctx).code
println(code)

final long value_0 = ((org.apache.spark.sql.delta.PartitionIdentityValueGenerator) references[0] /* generator */).next();