StateStoreProvider¶
StateStoreProvider
is the <
Note
StateStoreProvider
utility uses spark.sql.streaming.stateStore.providerClass internal configuration property for the name of the class of the default <
[[implementations]] NOTE: HDFSBackedStateStoreProvider is the default and only known StateStoreProvider
in Spark Structured Streaming.
[[contract]] .StateStoreProvider Contract [cols="30m,70",options="header",width="100%"] |=== | Method | Description
| close a| [[close]]
[source, scala]¶
close(): Unit¶
Closes the state store provider
Used exclusively when StateStore
helper object is requested to unload a state store provider
| doMaintenance a| [[doMaintenance]]
[source, scala]¶
doMaintenance(): Unit = {}¶
Optional state maintenance
Used exclusively when StateStore
utility is requested to perform maintenance of registered state store providers (on a separate MaintenanceTask daemon thread)
| getStore a| [[getStore]]
[source, scala]¶
getStore( version: Long): StateStore
Finds the StateStore for the specified version
Used exclusively when StateStore
utility is requested to look up the StateStore by a given provider ID
| init a| [[init]]
[source, scala]¶
init( stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, keyIndexOrdinal: Option[Int], storeConfs: StateStoreConf, hadoopConf: Configuration): Unit
Initializes the state store provider
Used exclusively when StateStoreProvider
helper object is requested to <StateStore
helper object is requested to retrieve a StateStore by ID and version)
| stateStoreId a| [[stateStoreId]]
[source, scala]¶
stateStoreId: StateStoreId¶
StateStoreId associated with the provider (at <
Used when:
-
HDFSBackedStateStore
is requested for the unique id -
HDFSBackedStateStoreProvider
is created and requested for the textual representation
| supportedCustomMetrics a| [[supportedCustomMetrics]]
[source, scala]¶
supportedCustomMetrics: Seq[StateStoreCustomMetric]¶
<
Used when:
-
StateStoreWriter
stateful physical operators are requested for the stateStoreCustomMetrics (when requested for the metrics and getProgress) -
HDFSBackedStateStore
is requested for the performance metrics
|===
=== [[createAndInit]] Creating and Initializing StateStoreProvider -- createAndInit
Object Method
[source, scala]¶
createAndInit( stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, indexOrdinal: Option[Int], storeConf: StateStoreConf, hadoopConf: Configuration): StateStoreProvider
createAndInit
creates a new <
createAndInit
requests the StateStoreProvider
to <
createAndInit
is used when StateStore
utility is requested for the StateStore by given provider ID and version.