Skip to content

StateStoreProvider

StateStoreProvider is the <> of <> that manage <> in Stateful Stream Processing (e.g. for persisting running aggregates in Streaming Aggregation) in stateful streaming queries.

Note

StateStoreProvider utility uses spark.sql.streaming.stateStore.providerClass internal configuration property for the name of the class of the default <>.

[[implementations]] NOTE: HDFSBackedStateStoreProvider is the default and only known StateStoreProvider in Spark Structured Streaming.

[[contract]] .StateStoreProvider Contract [cols="30m,70",options="header",width="100%"] |=== | Method | Description

| close a| [[close]]

[source, scala]

close(): Unit

Closes the state store provider

Used exclusively when StateStore helper object is requested to unload a state store provider

| doMaintenance a| [[doMaintenance]]

[source, scala]

doMaintenance(): Unit = {}

Optional state maintenance

Used exclusively when StateStore utility is requested to perform maintenance of registered state store providers (on a separate MaintenanceTask daemon thread)

| getStore a| [[getStore]]

[source, scala]

getStore( version: Long): StateStore


Finds the StateStore for the specified version

Used exclusively when StateStore utility is requested to look up the StateStore by a given provider ID

| init a| [[init]]

[source, scala]

init( stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, keyIndexOrdinal: Option[Int], storeConfs: StateStoreConf, hadoopConf: Configuration): Unit


Initializes the state store provider

Used exclusively when StateStoreProvider helper object is requested to <> for a given StateStoreId (when StateStore helper object is requested to retrieve a StateStore by ID and version)

| stateStoreId a| [[stateStoreId]]

[source, scala]

stateStoreId: StateStoreId

StateStoreId associated with the provider (at <>)

Used when:

| supportedCustomMetrics a| [[supportedCustomMetrics]]

[source, scala]

supportedCustomMetrics: Seq[StateStoreCustomMetric]

<> of the state store provider

Used when:

|===

=== [[createAndInit]] Creating and Initializing StateStoreProvider -- createAndInit Object Method

[source, scala]

createAndInit( stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, indexOrdinal: Option[Int], storeConf: StateStoreConf, hadoopConf: Configuration): StateStoreProvider


createAndInit creates a new <> (per spark.sql.streaming.stateStore.providerClass internal configuration property).

createAndInit requests the StateStoreProvider to <>.

createAndInit is used when StateStore utility is requested for the StateStore by given provider ID and version.