UserDefinedPythonFunction¶

Creating Instance¶

UserDefinedPythonFunction takes the following to be created:

• Name
• PythonFunction
• DataType (Spark SQL)
• Python Eval Type
• udfDeterministic flag

UserDefinedPythonFunction is created when:

• SparkConnectPlanner (Spark Connect) is requested to handleRegisterPythonUDF
• UserDefinedFunction (pyspark/sql/udf.py) is requested to _create_judf

Creating PythonUDF¶

builder(
  e: Seq[Expression]): Expression

builder creates a PythonUDF (for all the arguments and the given children expressions).

builder is used when:

• UDFRegistration is requested to register a Python UDF (Spark SQL)
• UserDefinedPythonFunction is requested to apply

Applying PythonUDF¶

apply(
  exprs: Column*): Column

apply creates a PythonUDF (for the input Column (Spark SQL) expressions) and wraps it up into a Column.

apply is used when:

• UDFRegistration is requested to register a Python UDF (Spark SQL)
• UserDefinedPythonFunction is requested to apply