GroupState¶

GroupState is an <> of <> (of type S) in Arbitrary Stateful Streaming Aggregation.

GroupState is used with the following KeyValueGroupedDataset operations:

• mapGroupsWithState
• flatMapGroupsWithState

GroupState is created separately for every aggregation key to hold a state as an aggregation state value.

[[contract]] .GroupState Contract [cols="30m,70",options="header",width="100%"] |=== | Method | Description

| exists a| [[exists]]

[source, scala]¶

exists: Boolean¶

Checks whether the state value exists or not

If not exists, <> throws a NoSuchElementException. Use <> instead.

| get a| [[get]]

[source, scala]¶

get: S¶

Gets the state value if it <> or throws a NoSuchElementException

| getCurrentProcessingTimeMs a| [[getCurrentProcessingTimeMs]]

[source, scala]¶

getCurrentProcessingTimeMs(): Long¶

Gets the current processing time (as milliseconds in epoch time)

| getCurrentWatermarkMs a| [[getCurrentWatermarkMs]]

[source, scala]¶

getCurrentWatermarkMs(): Long¶

Gets the current event time watermark (as milliseconds in epoch time)

| getOption a| [[getOption]]

[source, scala]¶

getOption: Option[S]¶

Gets the state value as a Scala Option (regardless whether it <> or not)

Used when:

• InputProcessor is requested to callFunctionAndUpdateState (when the row iterator is consumed and a state value has been updated, removed or timeout changed)

• GroupStateImpl is requested for the textual representation

| hasTimedOut a| [[hasTimedOut]]

[source, scala]¶

hasTimedOut: Boolean¶

Whether the state (for a given key) has timed out or not.

Can only be true when timeouts are enabled using <>

| remove a| [[remove]]

[source, scala]¶

remove(): Unit¶

Removes the state

| setTimeoutDuration a| [[setTimeoutDuration]]

[source, scala]¶

setTimeoutDuration(durationMs: Long): Unit setTimeoutDuration(duration: String): Unit

Specifies the timeout duration for the state key (in millis or as a string, e.g. "10 seconds", "1 hour") for GroupStateTimeout.ProcessingTimeTimeout

| setTimeoutTimestamp a| [[setTimeoutTimestamp]]

[source, scala]¶

setTimeoutTimestamp(timestamp: java.sql.Date): Unit setTimeoutTimestamp( timestamp: java.sql.Date, additionalDuration: String): Unit setTimeoutTimestamp(timestampMs: Long): Unit setTimeoutTimestamp( timestampMs: Long, additionalDuration: String): Unit

Specifies the timeout timestamp for the state key for GroupStateTimeout.EventTimeTimeout

| update a| [[update]]

[source, scala]¶

update(newState: S): Unit¶

Updates the state (sets the state to a new value)

|===

[[implementations]] GroupStateImpl is the default and only known implementation of the <> in Spark Structured Streaming.