StateStoreProviderId¶

[[creating-instance]] StateStoreProviderId is a unique identifier of a StateStoreProvider with the following properties:

• [[storeId]] StateStoreId
• [[queryRunId]] Run ID of a streaming query (java.util.UUID)

In other words, StateStoreProviderId is a <> with the <> that is different every restart.

StateStoreProviderId is used by the following execution components:

• StateStoreCoordinator to track the executors of state store providers (on the driver)

• StateStore object to manage state store providers (on executors)

StateStoreProviderId is <> (directly or using <> factory method) when:

• StateStoreRDD is requested for the placement preferences of a partition and to compute a partition

• StateStoreAwareZipPartitionsRDD is requested for the preferred locations of a partition

• StateStoreHandler is requested to look up a state store

Creating StateStoreProviderId¶

apply(
  stateInfo: StatefulOperatorStateInfo,
  partitionIndex: Int,
  storeName: String): StateStoreProviderId

apply simply creates a <> for the StatefulOperatorStateInfo, the partition and the store name.

Internally, apply requests the StatefulOperatorStateInfo for the checkpoint directory (checkpointLocation) and the stateful operator ID and creates a new StateStoreId (with the partitionIndex and storeName).

In the end, apply requests the StatefulOperatorStateInfo for the run ID of a streaming query and creates a <> (together with the run ID).

apply is used when:

• StateStoreAwareZipPartitionsRDD is requested for the preferred locations of a partition

• StateStoreHandler is requested to look up a state store