Skip to content


StateOperatorProgress is a progress of (updates made to) all the stateful operators in a micro-batch of a StreamingQuery.

StateOperatorProgress can be read in JSON format using jsonValue.

Creating Instance

StateOperatorProgress takes the following to be created:

  • Operator Name
  • numRowsTotal
  • numRowsUpdated
  • allUpdatesTimeMs
  • numRowsRemoved
  • allRemovalsTimeMs
  • commitTimeMs
  • memoryUsedBytes
  • numRowsDroppedByWatermark
  • numShufflePartitions
  • numStateStoreInstances
  • Custom Metrics

StateOperatorProgress is created when:

  • StateStoreWriter is requested to report progress
  • StateOperatorProgress is requested for a copy


numRowsTotal is the value of numTotalStateRows metric of a StateStoreWriter physical operator (when requested to get progress).


numRowsUpdated is the value of numUpdatedStateRows metric of a StateStoreWriter physical operator (when requested to get progress).

numRowsUpdated is updated when SessionWindowStateStoreSaveExec operator is requested for a StateOperatorProgress.

numRowsUpdated is displayed in Structured Streaming UI as Aggregated Number Of Updated State Rows.

Custom Metrics

customMetrics: Map[String, Long]

StateOperatorProgress can be given a collection of custom metrics (of the stateful operator it reports progress of). There are no metrics defined by default.

The custom metrics are stateStoreCustomMetrics and statefulOperatorCustomMetrics

Any custom metric included in spark.sql.streaming.ui.enabledCustomMetricList is displayed in Structured Streaming UI.

Included in jsonValue


  newNumRowsUpdated: Long,
  newNumRowsDroppedByWatermark: Long): StateOperatorProgress

copy creates a copy of this StateOperatorProgress with the numRowsUpdated and numRowsDroppedByWatermark metrics updated.

copy is used when:


jsonValue: JValue


jsonValue is used when: