Columnar Execution¶
Columnar Execution (Columnar Processing) uses the following:
- ColumnarRule
- ApplyColumnarRulesAndInsertTransitions physical optimization
- ColumnarToRowExec and ColumnarToRowExec physical operators
Physical operators that want to participate in Columnar Execution are expected to override supportsColumnar method.
Columnar Execution was introduced to Apache Spark 3.0.0 as SPARK-27396.
Whole-Stage Java Code Generation¶
Columnar Execution is similar and a kind of "opposite" at the same time to Whole-Stage Java Code Generation (which is row-based). It is assumed that if a plan supports columnar execution, it can't support whole-stage-codegen at the same time (see the comment in the source code).