Skip to content

ExtractPythonUDFs Logical Optimization

ExtractPythonUDFs is a logical optimization (Rule[LogicalPlan]) that "executes" (replaces) PythonUDFs as BaseEvalPythons.

Executing Rule

Rule 
apply(
  plan: LogicalPlan): LogicalPlan

apply is part of the Rule abstraction.

apply finds operators with PYTHON_UDF tree pattern (with the children first and this operator next) and extract from the given LogicalPlan.

Skips correlated subqueries

apply skips correlated subqueries.

extract

extract(
  plan: LogicalPlan): LogicalPlan

extract collects PythonUDFs (from the given LogicalPlan recursively) and rewrites them all to a single-type BaseEvalPython based on a PythonUDF eval type.

BaseEvalPython PythonUDF Eval Type
ArrowEvalPython SQL_SCALAR_PANDAS_UDF or SQL_SCALAR_PANDAS_ITER_UDF
BatchEvalPython SQL_BATCHED_UDF