SourceProgress¶

SourceProgress is a Serializable representation of progress of a SparkDataStream (in the execution of a StreamingQuery during a trigger).

Creating Instance¶

SourceProgress takes the following to be created:

• description
• startOffset
• endOffset
• latestOffset
• numInputRows
• inputRowsPerSecond
• processedRowsPerSecond
• Metrics

SourceProgress is created when:

• ProgressReporter is requested to finishTrigger (and update stream progress with a new StreamingQueryProgress)

Metrics¶

metrics: Map[String, String]

SourceProgress is given a metrics that is the metrics for this progress of ReportsSourceMetrics data stream.

metrics is used in jsonValue.

Converting to JSON Representation¶

jsonValue: JValue

jsonValue converts this SourceProgress to JSON representation.

jsonValue is used when:

• StreamingQueryProgress is requested for jsonValue
• SourceProgress is requested to json, prettyJson