Columnar Execution (Columnar Processing) is based on the following:
- ApplyColumnarRulesAndInsertTransitions physical optimization
- ColumnarToRowExec physical operator
Physical operators that want to participate in Columnar Execution are expected to override supportsColumnar method.
Columnar Execution was introduced to Apache Spark 3.0.0 as SPARK-27396.
Whole-Stage Java Code Generation¶
Columnar Execution is similar and a kind of "opposite" at the same time to Whole-Stage Java Code Generation (which is row-based). It is assumed that if a plan supports columnar execution, it can't support whole-stage-codegen at the same time (see the comment in the source code).