PythonFunction¶

PythonFunction is an abstraction of the metadata of a Python function of a PythonUDF.

PythonFunction is executed in a BasePythonRunner.

PythonFunction is used to create the following:

• PythonRDD
• PythonRunner
• PythonForeachWriter
• UserDefinedPythonFunction

Contract (Subset)¶

accumulator¶

accumulator: PythonAccumulatorV2

PythonAccumulatorV2

Used when:

• BasePythonRunner is created

broadcastVars¶

broadcastVars: JList[Broadcast[PythonBroadcast]]

A collection of broadcast variables (Spark Core) with a PythonBroadcast

Used when:

• WriterThread is created

command¶

command: Seq[Byte]

Used when:

• PythonRunner is requested to newWriterThread
• UDFRegistration is requested to register a Python UDF (for logging purposes only)
• PythonUDFRunner is requested to writeUDFs

Implementations¶

• SimplePythonFunction