Nondeterministic Expressions¶

Nondeterministic is an extension of the Expression abstraction for non-deterministic and non-foldable expressions.

Nondeterministic expression should be initialized (with the partition ID) before evaluation.

Contract¶

evalInternal¶

evalInternal(
  input: InternalRow): Any

Used when:

• Nondeterministic is requested to eval

initializeInternal¶

initializeInternal(
  partitionIndex: Int): Unit

Used when:

• Nondeterministic is requested to initialize

Implementations¶

• CallMethodViaReflection
• CurrentBatchTimestamp
• InputFileBlockLength
• InputFileBlockStart
• InputFileName
• SparkPartitionID
• Stateful

Review Me¶

NOTE: Nondeterministic expressions are the target of PullOutNondeterministic logical plan rule.

=== [[initialize]] Initializing Expression -- initialize Method

[source, scala]¶

initialize(partitionIndex: Int): Unit¶

Internally, initialize <> itself (with the input partition index) and turns the internal <> flag on.

initialize is used when InterpretedProjection and InterpretedMutableProjection are requested to initialize themselves.

=== [[eval]] Evaluating Expression -- eval Method

[source, scala]¶

eval(input: InternalRow): Any¶

eval is part of the Expression abstraction.

eval is just a wrapper of <> that makes sure that <> has already been executed (and so the expression is initialized).

Internally, eval makes sure that the expression was <> and calls <>.

eval reports a IllegalArgumentException exception when the internal <> flag is off, i.e. <> has not yet been executed.

requirement failed: Nondeterministic expression [name] should be initialized before eval.