Skip to content

StateStoreRDD

StateStoreRDD is a BaseStateStoreRDD.

StateStoreRDD, Physical and Logical Plans, and operators

Creating Instance

StateStoreRDD takes the following to be created:

  • Data RDD (RDD[T] to update the aggregates in a state store)
  • Store update function ((StateStore, Iterator[T]) => Iterator[U] where T is the type of rows in the data RDD)
  • Checkpoint directory
  • Run ID of the streaming query
  • Operator ID
  • Store Version
  • Key Schema
  • Value Schema
  • numColsPrefixKey
  • SessionState
  • StateStoreCoordinatorRef
  • Extra Options

StateStoreRDD is created when:

numColsPrefixKey

StateStoreRDD is given numColsPrefixKey when created (that is passed in directly from StateStoreOps.mapPartitionsWithStateStore).

numColsPrefixKey is used to look up a StateStore while computing a partition.