SimplePythonFunction¶

SimplePythonFunction is a PythonFunction.

Creating Instance¶

SimplePythonFunction takes the following to be created:

• Command (byte array)
• Environment Variables
• Python Includes
• Python Executable
• Python Version
• Broadcasts of PythonBroadcasts
• PythonAccumulatorV2

SimplePythonFunction is created when:

• SparkConnectPlanner is requested to transformPythonFunction
• pyspark.rdd (Python module) is requested to _wrap_function
• pyspark.sql.udf (Python module) is requested to _wrap_function

Python Executable¶

SimplePythonFunction is given the Python Executable when created.

The Python Executable is controlled by PYSPARK_PYTHON environment variable (in PySpark) or PYSPARK_DRIVER_PYTHON (in PySpark Connect).