Skip to content

StreamProgress

StreamProgress is an immutable Map (Scala) of Offsets by SparkDataStream (Map[SparkDataStream, OffsetV2])

class StreamProgress(...)
extends Map[SparkDataStream, OffsetV2]

In other words, StreamProgress represents source data streams of a streaming query with their position.

Creating Instance

StreamProgress takes the following to be created:

  • Base Map (default: empty)

StreamProgress is created when:

toOffsetSeq

toOffsetSeq(
  source: Seq[SparkDataStream],
  metadata: OffsetSeqMetadata): OffsetSeq

toOffsetSeq creates an OffsetSeq.


toOffsetSeq is used when:

  • StreamExecution is requested to runStream (to create a StreamingQueryException when a streaming query fails)