Skip to content

MonotonicallyIncreasingID

MonotonicallyIncreasingID is a non-deterministic leaf expression that represents monotonically_increasing_id standard and SQL functions in logical query plans.

MonotonicallyIncreasingID supports code-generated and interpreted execution modes.

Result DataType

dataType: DataType

dataType is always LongType

dataType is part of the Expression abstraction.

Never Nullable

nullable: Boolean

nullable is always false.

nullable is part of the Expression abstraction.

Initialization

initializeInternal(
  partitionIndex: Int): Unit

initializeInternal initializes the following internal registries:

val partitionIndex = 1
val partitionMask = partitionIndex.toLong << 33
scala> println(partitionMask.toBinaryString)
1000000000000000000000000000000000

initializeInternal is part of the Nondeterministic abstraction.

Interpreted Expression Evaluation

evalInternal(
  input: InternalRow): Long

evalInternal increments the count internal counter.

evalInternal increments the partitionMask internal registry by the previous count.

evalInternal is part of the Nondeterministic abstraction.

Code-Generated Expression Evaluation

doGenCode(
  ctx: CodegenContext,
  ev: ExprCode): ExprCode

doGenCode is part of the Expression abstraction.


doGenCode requests the CodegenContext to add a mutable state as count name and long Java type.

doGenCode requests the CodegenContext to add an immutable state (unless exists already) as partitionMask name and long Java type.

doGenCode requests the CodegenContext to addPartitionInitializationStatement with [countTerm] = 0L; statement.

doGenCode requests the CodegenContext to addPartitionInitializationStatement with [partitionMaskTerm] = ((long) partitionIndex) << 33; statement.

In the end, doGenCode returns the input ExprCode with the code as follows and isNull property disabled (false):

final [dataType] [value] = [partitionMaskTerm] + [countTerm];
      [countTerm]++;

import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
val monotonicallyIncreasingID = MonotonicallyIncreasingID()

// doGenCode is used when Expression.genCode is executed

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext
val code = monotonicallyIncreasingID.genCode(ctx).code
scala> println(code)
final long value_0 = partitionMask + count_0;
      count_0++;

Stateful

MonotonicallyIncreasingID is a Stateful.

Back to top