KeyWithIndexToValueStore¶

KeyWithIndexToValueStore is a StateStoreHandler (of KeyWithIndexToValueType) for SymmetricHashJoinStateManager to manage a <>.

.KeyToNumValuesStore, KeyWithIndexToValueStore and Stream-Stream Join image::images/KeyToNumValuesStore-KeyWithIndexToValueStore.png[align="center"]

[[stateStore]] As a StateStoreHandler, KeyWithIndexToValueStore manages a state store (that is loaded) for keys and values per the <> and input values schemas, respectively.

[[keyWithIndexSchema]] KeyWithIndexToValueStore uses a schema (for the <>) that is the key schema (of the parent SymmetricHashJoinStateManager) with an extra field index of type long.

[[logging]] [TIP] ==== Enable ALL logging level for org.apache.spark.sql.execution.streaming.state.SymmetricHashJoinStateManager$KeyWithIndexToValueStore to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.execution.streaming.state.SymmetricHashJoinStateManager$KeyWithIndexToValueStore=ALL

Refer to <>.¶

=== [[get]] Looking Up State Row For Given Key and Index -- get Method

[source, scala]¶

get( key: UnsafeRow, valueIndex: Long): UnsafeRow

get simply requests the internal state store to look up the value for the given <>.

get is used when SymmetricHashJoinStateManager is requested to removeByValueCondition.

=== [[getAll]] Retrieving (Given Number of) Values for Key -- getAll Method

[source, scala]¶

getAll( key: UnsafeRow, numValues: Long): Iterator[KeyWithIndexAndValue]

getAll...FIXME

getAll is used when SymmetricHashJoinStateManager is requested to get values for a given key and removeByKeyCondition.

=== [[put]] Storing State Row For Given Key and Index -- put Method

[source, scala]¶

put( key: UnsafeRow, valueIndex: Long, value: UnsafeRow): Unit

put...FIXME

put is used when SymmetricHashJoinStateManager is requested to append a new value to a given key and removeByKeyCondition.

=== [[remove]] remove Method

[source, scala]¶

remove( key: UnsafeRow, valueIndex: Long): Unit

remove...FIXME

remove is used when SymmetricHashJoinStateManager is requested to removeByKeyCondition and removeByValueCondition.

=== [[keyWithIndexRow]] keyWithIndexRow Internal Method

[source, scala]¶

keyWithIndexRow( key: UnsafeRow, valueIndex: Long): UnsafeRow

keyWithIndexRow uses the <> to generate an UnsafeRow for the key and sets the valueIndex at the <> position.

NOTE: keyWithIndexRow is used when KeyWithIndexToValueStore is requested to <>, <>, <>, <> and <>.

=== [[removeAllValues]] removeAllValues Method

[source, scala]¶

removeAllValues( key: UnsafeRow, numValues: Long): Unit

removeAllValues...FIXME

NOTE: removeAllValues does not seem to be used at all.

=== [[iterator]] iterator Method

[source, scala]¶

iterator: Iterator[KeyWithIndexAndValue]¶

iterator...FIXME

NOTE: iterator does not seem to be used at all.

Internal Properties¶

[cols="30m,70",options="header",width="100%"] |=== | Name | Description

| indexOrdinalInKeyWithIndexRow a| [[indexOrdinalInKeyWithIndexRow]] Position of the index in the key row (which corresponds to the number of the key attributes)

Used exclusively in the <>

| keyWithIndexExprs a| [[keyWithIndexExprs]] keyAttributes with Literal(1L) expression appended

Used exclusively for the <> projection

| keyWithIndexRowGenerator a| [[keyWithIndexRowGenerator]] UnsafeProjection for the <> bound to the keyAttributes

Used exclusively in <>

|===