SparkUserDefinedFunction¶

SparkUserDefinedFunction is a UserDefinedFunction that uses ScalaUDF for execution.

SparkUserDefinedFunction is created using udf standard function (among the other less interesting means).

Creating Instance¶

SparkUserDefinedFunction takes the following to be created:

• Scala Function
• DataType
• Input ExpressionEncoders
• Output ExpressionEncoder
• Name
• nullable flag (default: true)
• deterministic flag (default: true)

SparkUserDefinedFunction is created when:

• FPGrowthModel (Spark MLlib) is requested to genericTransform
• udf standard function is used
• UDFRegistration is requested to register a named user-defined function

Creating Column (for Function Execution)¶

apply(
  exprs: Column*): Column

apply is part of the UserDefinedFunction abstraction.

apply creates a Column with a ScalaUDF (with the given exprs).

Creating ScalaUDF¶

createScalaUDF(
  exprs: Seq[Expression]): ScalaUDF

createScalaUDF creates a ScalaUDF expression.

createScalaUDF is used when:

• UDFRegistration is requested to register a named user-defined function
• SparkUserDefinedFunction is requested to create a Column (for function execution)