StateStoreProvider¶

StateStoreProvider is an abstraction of StateStore providers that manage state stores (state data) in Stateful Stream Processing.

A concrete StateStoreProvider is selected based on spark.sql.streaming.stateStore.providerClass configuration property.

Contract¶

Looking Up StateStore by Version¶

getStore(
  version: Long): StateStore

StateStore for the given version

See:

• HDFSBackedStateStoreProvider
• RocksDBStateStoreProvider

Used when:

• StateStoreProvider is requested for a ReadStateStore by version
• StateStore utility is used to get a StateStore by provider id and version

Initialization¶

init(
  stateStoreId: StateStoreId,
  keySchema: StructType,
  valueSchema: StructType,
  numColsPrefixKey: Int,
  storeConfs: StateStoreConf,
  hadoopConf: Configuration): Unit

Initializes a StateStoreProvider with the following:

• StateStoreId
• Key and value schema
• StateStoreConf

See:

• HDFSBackedStateStoreProvider
• RocksDBStateStoreProvider

Used when:

• StateStoreProvider utility is used to create and initialize a StateStoreProvider

Supported Custom Metrics¶

supportedCustomMetrics: Seq[StateStoreCustomMetric]

StateStoreCustomMetrics (e.g., to report in web UI or StateOperatorProgress)

Default: empty

See:

• RocksDBStateStoreProvider
• HDFSBackedStateStoreProvider

Used when:

• StateStoreWriter is requested for the stateStoreCustomMetrics

Implementations¶

• HDFSBackedStateStoreProvider
• RocksDBStateStoreProvider

Creating StateStoreProvider¶

create(
  providerClassName: String): StateStoreProvider

create creates a StateStoreProvider based on the fully-qualified class name (providerClassName).

create is used when:

• StateStoreWriter physical operator is requested to stateStoreCustomMetrics
• StateStoreProvider utility is used to create and initialize a StateStoreProvider
• StreamingQueryStatisticsPage is requested for the supportedCustomMetrics

Creating and Initializing StateStoreProvider¶

createAndInit(
  providerId: StateStoreProviderId,
  keySchema: StructType,
  valueSchema: StructType,
  numColsPrefixKey: Int,
  storeConf: StateStoreConf,
  hadoopConf: Configuration): StateStoreProvider

createAndInit creates a StateStoreProvider based on spark.sql.streaming.stateStore.providerClass.

In the end, createAndInit requests the StateStoreProvider to initialize.

createAndInit is used when:

• StateStore utility is used to look up a StateStore by StateStoreProviderId

Looking Up ReadStateStore by Version¶

getReadStore(
  version: Long): ReadStateStore

getReadStore creates a WrappedReadStateStore for the StateStore for the given version.

getReadStore is used when:

• StateStore utility is used to get a ReadStateStore