Skip to content

Columnar Execution

Columnar Execution (Columnar Processing) is based on the following:

Physical operators that want to participate in Columnar Execution are expected to override supportsColumnar method.

Columnar Execution was introduced to Apache Spark 3.0.0 as SPARK-27396.

Whole-Stage Java Code Generation

Columnar Execution is similar and a kind of "opposite" at the same time to Whole-Stage Java Code Generation (which is row-based). It is assumed that if a plan supports columnar execution, it can't support whole-stage-codegen at the same time (see the comment in the source code).

References

Articles