StreamProgress¶

StreamProgress is an immutable Map (Scala) of Offsets by SparkDataStream (Map[SparkDataStream, OffsetV2])

class StreamProgress(...)
extends Map[SparkDataStream, OffsetV2]

In other words, StreamProgress represents source data streams of a streaming query with their position.

Creating Instance¶

StreamProgress takes the following to be created:

• Base Map (default: empty)

StreamProgress is created when:

• OffsetSeq is requested to toStreamProgress
• StreamExecution is created (and creates committed, available, latest offset trackers)
• StreamProgress is requested to ++

toOffsetSeq¶

toOffsetSeq(
  source: Seq[SparkDataStream],
  metadata: OffsetSeqMetadata): OffsetSeq

toOffsetSeq creates an OffsetSeq.

toOffsetSeq is used when:

• StreamExecution is requested to runStream (to create a StreamingQueryException when a streaming query fails)