ColumnarToRowExec Physical Operator¶
ColumnarToRowExec is a ColumnarToRowTransition unary physical operator to translate an RDD of ColumnarBatches into an RDD of InternalRows in Columnar Processing.
ColumnarToRowExec supports Whole-Stage Java Code Generation.
ColumnarToRowExec takes the following to be created:
- Child physical operator
ColumnarToRowExec is created when:
- ApplyColumnarRulesAndInsertTransitions physical optimization is executed
number of input batches¶
Number of input ColumnarBatches across all partitions (from columnar execution of the child physical operator that produces
RDD[ColumnarBatch] and hence RDD partitions with rows "compressed" into
number of output rows¶
Executing Physical Operator¶
doExecute is part of the SparkPlan abstraction.
doExecute requests the child physical operator to executeColumnar (which is valid since it does support columnar processing) and
RDD.mapPartitionsInternal over partitions of ColumnarBatches (
Iterator[ColumnarBatch]) to "unpack" / "uncompress" them to InternalRows.
inputRDDs is part of the CodegenSupport abstraction.
canCheckLimitNotReached is part of the CodegenSupport abstraction.
canCheckLimitNotReached is always enabled (