Skip to content

Nondeterministic Expressions

Nondeterministic is an extension of the Expression abstraction for non-deterministic and non-foldable expressions.

Nondeterministic expression should be initialized (with the partition ID) before evaluation.

Contract

evalInternal

evalInternal(
  input: InternalRow): Any

Used when:

  • Nondeterministic is requested to eval

initializeInternal

initializeInternal(
  partitionIndex: Int): Unit

Used when:

Implementations

  • CallMethodViaReflection
  • CurrentBatchTimestamp
  • InputFileBlockLength
  • InputFileBlockStart
  • InputFileName
  • SparkPartitionID
  • Stateful

Review Me

NOTE: Nondeterministic expressions are the target of PullOutNondeterministic logical plan rule.

=== [[initialize]] Initializing Expression -- initialize Method

[source, scala]

initialize(partitionIndex: Int): Unit

Internally, initialize <> itself (with the input partition index) and turns the internal <> flag on.

initialize is used when InterpretedProjection and InterpretedMutableProjection are requested to initialize themselves.

=== [[eval]] Evaluating Expression -- eval Method

[source, scala]

eval(input: InternalRow): Any

eval is part of the Expression abstraction.

eval is just a wrapper of <> that makes sure that <> has already been executed (and so the expression is initialized).

Internally, eval makes sure that the expression was <> and calls <>.

eval reports a IllegalArgumentException exception when the internal <> flag is off, i.e. <> has not yet been executed.

requirement failed: Nondeterministic expression [name] should be initialized before eval.
Back to top