AggregationIterators¶

AggregationIterator is an abstraction of aggregation iterators of UnsafeRows.

abstract class AggregationIterator(...)
extends Iterator[UnsafeRow]

From scala.collection.Iterator:

Iterators are data structures that allow to iterate over a sequence of elements. They have a hasNext method for checking if there is a next element available, and a next method which returns the next element and discards it from the iterator.

Implementations¶

• ObjectAggregationIterator
• SortBasedAggregationIterator
• TungstenAggregationIterator

Creating Instance¶

AggregationIterator takes the following to be created:

• Partition ID
• Grouping NamedExpressions
• Input Attributes
• AggregateExpressions
• Aggregate Attributes
• Initial input buffer offset
• Result NamedExpressions
• Function to create a new MutableProjection given expressions and attributes ((Seq[Expression], Seq[Attribute]) => MutableProjection)

AggregateModes¶

When created, AggregationIterator makes sure that there are at most 2 distinct AggregateModes of the AggregateExpressions.

The AggregateModes have to be a subset of the following mode pairs:

• Partial and PartialMerge
• Final and Complete

AggregateFunctions¶

aggregateFunctions: Array[AggregateFunction]

When created, AggregationIterator initializes AggregateFunctions in the aggregateExpressions (with initialInputBufferOffset).

initializeAggregateFunctions¶

initializeAggregateFunctions(
  expressions: Seq[AggregateExpression],
  startingInputBufferOffset: Int): Array[AggregateFunction]

initializeAggregateFunctions...FIXME

initializeAggregateFunctions is used when:

• AggregationIterator is requested for the aggregateFunctions
• ObjectAggregationIterator is requested for the mergeAggregationBuffers
• TungstenAggregationIterator is requested to switchToSortBasedAggregation

generateProcessRow¶

generateProcessRow(
  expressions: Seq[AggregateExpression],
  functions: Seq[AggregateFunction],
  inputAttributes: Seq[Attribute]): (InternalRow, InternalRow) => Unit

generateProcessRow...FIXME

generateProcessRow is used when:

• AggregationIterator is requested for the processRow function
• ObjectAggregationIterator is requested for the mergeAggregationBuffers function
• TungstenAggregationIterator is requested to switchToSortBasedAggregation

generateOutput¶

generateOutput: (UnsafeRow, InternalRow) => UnsafeRow

When created, AggregationIterator creates a ResultProjection function.

generateOutput is used when:

• ObjectAggregationIterator is requested for the next element and to outputForEmptyGroupingKeyWithoutInput
• SortBasedAggregationIterator is requested for the next element and to outputForEmptyGroupingKeyWithoutInput
• TungstenAggregationIterator is requested for the next element and to outputForEmptyGroupingKeyWithoutInput

generateResultProjection¶

generateResultProjection(): (UnsafeRow, InternalRow) => UnsafeRow

generateResultProjection...FIXME

initializeBuffer¶

initializeBuffer(
  buffer: InternalRow): Unit

initializeBuffer...FIXME

initializeBuffer is used when:

• SortBasedAggregationIterator is requested to newBuffer, initialize, next and outputForEmptyGroupingKeyWithoutInput