StateStoreProvider¶

StateStoreProvider is the <> of <> that manage <> in <> (e.g. for persisting running aggregates in Streaming Aggregation) in stateful streaming queries.

Note

StateStoreProvider utility uses spark.sql.streaming.stateStore.providerClass internal configuration property for the name of the class of the default <>.

[[implementations]] NOTE: HDFSBackedStateStoreProvider is the default and only known StateStoreProvider in Spark Structured Streaming.

[[contract]] .StateStoreProvider Contract [cols="30m,70",options="header",width="100%"] |=== | Method | Description

| close a| [[close]]

[source, scala]¶

close(): Unit¶

Closes the state store provider

Used exclusively when StateStore helper object is requested to unload a state store provider

| doMaintenance a| [[doMaintenance]]

[source, scala]¶

doMaintenance(): Unit = {}¶

Optional state maintenance

Used exclusively when StateStore utility is requested to perform maintenance of registered state store providers (on a separate MaintenanceTask daemon thread)

| getStore a| [[getStore]]

[source, scala]¶

getStore( version: Long): StateStore

Finds the StateStore for the specified version

Used exclusively when StateStore utility is requested to look up the StateStore by a given provider ID

| init a| [[init]]

[source, scala]¶

init( stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, keyIndexOrdinal: Option[Int], storeConfs: StateStoreConf, hadoopConf: Configuration): Unit

Initializes the state store provider

Used exclusively when StateStoreProvider helper object is requested to <> for a given StateStoreId (when StateStore helper object is requested to retrieve a StateStore by ID and version)

| stateStoreId a| [[stateStoreId]]

[source, scala]¶

stateStoreId: StateStoreId¶

StateStoreId associated with the provider (at <>)

Used when:

HDFSBackedStateStore is requested for the unique id
HDFSBackedStateStoreProvider is created and requested for the textual representation

| supportedCustomMetrics a| [[supportedCustomMetrics]]

[source, scala]¶

supportedCustomMetrics: Seq[StateStoreCustomMetric]¶

<> of the state store provider

Used when:

StateStoreWriter stateful physical operators are requested for the stateStoreCustomMetrics (when requested for the metrics and getProgress)
HDFSBackedStateStore is requested for the performance metrics

|===

=== [[createAndInit]] Creating and Initializing StateStoreProvider -- createAndInit Object Method

[source, scala]¶

createAndInit( stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, indexOrdinal: Option[Int], storeConf: StateStoreConf, hadoopConf: Configuration): StateStoreProvider

createAndInit creates a new <> (per spark.sql.streaming.stateStore.providerClass internal configuration property).

createAndInit requests the StateStoreProvider to <>.

createAndInit is used when StateStore utility is requested for the StateStore by given provider ID and version.