UserDefinedAggregator¶

UserDefinedAggregator[IN, BUF, OUT] is a UserDefinedFunction.

Creating Instance¶

UserDefinedAggregator takes the following to be created:

• Aggregator[IN, BUF, OUT]
• Encoder[IN] (assumed a ExpressionEncoder)
• Name (optional)
• nullable flag (default: true)
• deterministic flag (default: true)

UserDefinedAggregator is created using udaf standard function.

Creating Column for UDAF Execution¶

apply(
  exprs: Column*): Column

apply creates a Column with an AggregateExpression with the following:

• ScalaAggregator aggregate function
• Complete aggregate mode
• isDistinct flag disabled (false)

apply is part of the UserDefinedFunction abstraction.

Creating ScalaAggregator¶

scalaAggregator(
  exprs: Seq[Expression]): ScalaAggregator[IN, BUF, OUT]

scalaAggregator assumes the following are all ExpressionEncoders:

1. Input Encoder
2. Buffer Encoder of the Aggregator

In the end, scalaAggregator creates a ScalaAggregator.

scalaAggregator is used when:

• UDFRegistration is requested to register a UserDefinedAggregator
• UserDefinedAggregator is requested to create a Column (for UDAF execution)