KeyToNumValuesStore¶

KeyToNumValuesStore is a StateStoreHandler (of KeyToNumValuesType) for SymmetricHashJoinStateManager to manage a <>.

[[stateStore]] As a StateStoreHandler, KeyToNumValuesStore manages a state store (that is loaded) with the join keys (per key schema) and their count (per <>).

[[longValueSchema]] KeyToNumValuesStore uses the schema for values in the <> with one field value (of type long) that is the number of value rows (count).

[[logging]] [TIP] ==== Enable ALL logging level for org.apache.spark.sql.execution.streaming.state.SymmetricHashJoinStateManager$KeyToNumValuesStore to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.execution.streaming.state.SymmetricHashJoinStateManager$KeyToNumValuesStore=ALL

Refer to <>.¶

=== [[get]] Looking Up Number Of Value Rows For Given Key (Value Count) -- get Method

[source, scala]¶

get(key: UnsafeRow): Long¶

get requests the <> for the value for the given key and returns the long value at 0th position (of the row found) or 0.

get is used when SymmetricHashJoinStateManager is requested for the values for a given key and append a new value to a given key.

=== [[put]] Storing Key Count For Given Key -- put Method

[source, scala]¶

put( key: UnsafeRow, numValues: Long): Unit

put stores the numValues at the 0th position (of the internal unsafe row) and requests the <> to store it with the given key.

put requires that the numValues count is greater than 0 (or throws an IllegalArgumentException).

put is used when SymmetricHashJoinStateManager is requested for the append a new value to a given key and updateNumValueForCurrentKey.

=== [[iterator]] All State Keys and Values -- iterator Method

[source, scala]¶

iterator: Iterator[KeyAndNumValues]¶

iterator simply requests the <> for all state keys and values.

iterator is used when SymmetricHashJoinStateManager is requested to removeByKeyCondition and removeByValueCondition.

=== [[remove]] Removing State Key -- remove Method

[source, scala]¶

remove(key: UnsafeRow): Unit¶

remove simply requests the <> to remove the given key.

remove is used when...FIXME