MonotonicallyIncreasingID Leaf Expression¶

MonotonicallyIncreasingID is a non-deterministic leaf expression that represents monotonically_increasing_id standard and SQL functions in logical query plans.

MonotonicallyIncreasingID supports code-generated and interpreted execution modes.

Result DataType¶

dataType: DataType

dataType is always LongType

dataType is part of the Expression abstraction.

Never Nullable¶

nullable: Boolean

nullable is always false.

nullable is part of the Expression abstraction.

Internal Initialize¶

initializeInternal(
  partitionIndex: Int): Unit

initializeInternal initializes the following internal registries:

• count to 0
• partitionMask as partitionIndex.toLong << 33

val partitionIndex = 1
val partitionMask = partitionIndex.toLong << 33
scala> println(partitionMask.toBinaryString)
1000000000000000000000000000000000

initializeInternal is part of the Nondeterministic abstraction.

Internal Interpreted Expression Evaluation¶

evalInternal(
  input: InternalRow): Long

evalInternal increments the count internal counter.

evalInternal increments the partitionMask internal registry by the previous count.

evalInternal is part of the Nondeterministic abstraction.

Code-Generated Expression Evaluation¶

doGenCode(
  ctx: CodegenContext,
  ev: ExprCode): ExprCode

doGenCode is part of the Expression abstraction.

doGenCode requests the CodegenContext to add a mutable state as count name and long Java type.

doGenCode requests the CodegenContext to add an immutable state (unless exists already) as partitionMask name and long Java type.

doGenCode requests the CodegenContext to addPartitionInitializationStatement with [countTerm] = 0L; statement.

doGenCode requests the CodegenContext to addPartitionInitializationStatement with [partitionMaskTerm] = ((long) partitionIndex) << 33; statement.

In the end, doGenCode returns the input ExprCode with the code as follows and isNull property disabled (false):

final [dataType] [value] = [partitionMaskTerm] + [countTerm];
      [countTerm]++;

import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
val monotonicallyIncreasingID = MonotonicallyIncreasingID()

// doGenCode is used when Expression.genCode is executed

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext
val code = monotonicallyIncreasingID.genCode(ctx).code

scala> println(code)
final long value_0 = partitionMask + count_0;
      count_0++;

Stateful¶

MonotonicallyIncreasingID is a Stateful.