ArrowPythonRunner¶
ArrowPythonRunner
is a pyspark runner with PythonArrowOutput
for...FIXME
Creating Instance¶
ArrowPythonRunner
takes the following to be created:
-
Seq[ChainedPythonFunctions]
- Eval Type
- Argument Offsets (
Array[Array[Int]]
) -
Schema
(Spark SQL) - TimeZone ID
- Configuration (
Map[String, String]
)
ArrowPythonRunner
is created when AggregateInPandasExec, ArrowEvalPythonExec
, FlatMapGroupsInPandasExec
, MapInPandasExec
, WindowInPandasExec
physical operators are executed.
Last update: 2021-03-03