StreamingAggregationStateManagerImplV2¶

StreamingAggregationStateManagerImplV2 is the default state manager for streaming aggregations (used when spark.sql.streaming.aggregation.stateFormatVersion configuration property is 2).

Creating Instance¶

StreamingAggregationStateManagerImplV2 (like the parent StreamingAggregationStateManagerBaseImpl) takes the following to be created:

[[keyExpressions]] Catalyst expressions for the keys (Seq[Attribute])
[[inputRowAttributes]] Catalyst expressions for the input rows (Seq[Attribute])

StreamingAggregationStateManagerImplV2 is created when StreamingAggregationStateManager is requested for a new StreamingAggregationStateManager.

Storing Row in State Store¶

put(
  store: StateStore,
  row: UnsafeRow): Unit

put is part of the StreamingAggregationStateManager abstraction.

put...FIXME

Getting Saved State for Non-Null Key from State Store¶

get(
  store: StateStore,
  key: UnsafeRow): UnsafeRow

get is part of the StreamingAggregationStateManager abstraction.

get requests the given StateStore for the current state value for the given key.

get returns null if the key could not be found in the state store. Otherwise, get restoreOriginalRow (for the key and the saved state).

=== [[restoreOriginalRow]] restoreOriginalRow Internal Method

[source, scala]¶

restoreOriginalRow(key: UnsafeRow, value: UnsafeRow): UnsafeRow restoreOriginalRow(rowPair: UnsafeRowPair): UnsafeRow

restoreOriginalRow...FIXME

NOTE: restoreOriginalRow is used when StreamingAggregationStateManagerImplV2 is requested to <>, <> and <>.

getStateValueSchema¶

getStateValueSchema: StructType

getStateValueSchema is part of the StreamingAggregationStateManager abstraction.

getStateValueSchema simply requests the valueExpressions for the schema.

iterator¶

iterator(
  store: StateStore): Iterator[UnsafeRowPair]

iterator is part of the StreamingAggregationStateManager abstraction.

iterator simply requests the input state store for the iterator that is mapped to an iterator of UnsafeRowPairs with the key (of the input UnsafeRowPair) and the value as a restored original row.

Note

scala.collection.Iterator is a data structure that allows to iterate over a sequence of elements that are usually fetched lazily (i.e. no elements are fetched from the underlying store until processed).

values¶

values(
  store: StateStore): Iterator[UnsafeRow]

values is part of the StreamingAggregationStateManager abstraction.

values...FIXME

Internal Properties¶

[cols="30m,70",options="header",width="100%"] |=== | Name | Description

| joiner | [[joiner]]

| keyValueJoinedExpressions a| [[keyValueJoinedExpressions]]

| needToProjectToRestoreValue a| [[needToProjectToRestoreValue]]

| restoreValueProjector a| [[restoreValueProjector]]

| valueExpressions a| [[valueExpressions]]

| valueProjector a| [[valueProjector]] |===