StreamingQueryProgress¶

StreamingQueryProgress is metadata of a single micro-batch (progress) of a StreamingQuery.

Creating Instance¶

StreamingQueryProgress takes the following to be created:

• Unique identifier
• Unique identifier of a query execution
• Name
• Batch Timestamp
• Unique ID of a micro-batch
• Batch Duration
• Durations of the internal phases (in ms)
• EventTime Statistics
• StateOperatorProgress for every stateful operator
• SourceProgress for every streaming source
• SinkProgress
• Observed Metrics

StreamingQueryProgress is created when:

• StreamExecution is requested to finish a trigger

Batch Timestamp¶

StreamingQueryProgress is given a timestamp when created.

The time when a trigger has started (in ISO8601 format).

Event Time Statistics¶

StreamingQueryProgress is given an Event Time Statistics when created.

inputRowsPerSecond¶

inputRowsPerSecond: Double

inputRowsPerSecond is the total (sum) of the inputRowsPerSeconds of all the SourceProgresses (of this single StreamingQueryProgress).

inputRowsPerSecond is used when:

• MetricsReporter is requested to register the inputRate-total metric
• StreamingQueryProgress is requested for jsonValue (for the inputRowsPerSecond field)
• StreamingQueryStatisticsPage is requested to display Input Rate timeline and histogram

Total Number of Input Rows¶

numInputRows: Long

numInputRows is a sum of the numInputRows of all the sources.

numInputRows is used when:

• StreamingQueryProgress is requested for jsonValue
• StreamingQueryStatisticsPage is requested to generateStatTable (for input-rows-histogram)

Last and Recent Progresses¶

Use lastProgress property of a StreamingQuery to access the most recent StreamingQueryProgress update.

val sq: StreamingQuery = ...
sq.lastProgress

Use recentProgress property of a StreamingQuery to access the most recent StreamingQueryProgress updates.

val sq: StreamingQuery = ...
sq.recentProgress

StreamingQueryListener¶

Use StreamingQueryListener to be notified about StreamingQueryProgress updates while a streaming query is executed.

prettyJson¶

prettyJson: String

prettyJson...FIXME

prettyJson is used for toString.

jsonValue¶

jsonValue: JValue

jsonValue...FIXME

jsonValue is used in json and prettyJson.