RatePerMicroBatchStream¶

Creating Instance¶

RatePerMicroBatchStream takes the following to be created:

RatePerMicroBatchStream is created when:

RatePerMicroBatchStream defines isTriggerAvailableNow flag to determine whether it is executed in Trigger.AvailableNow mode or not (based on prepareForTriggerAvailableNow).

By default, isTriggerAvailableNow flag is off (false).

isTriggerAvailableNow is used when:

prepareForTriggerAvailableNow(): Unit

prepareForTriggerAvailableNow is part of the SupportsTriggerAvailableNow abstraction.

prepareForTriggerAvailableNow turns the isTriggerAvailableNow flag on (for Trigger.AvailableNow mode).

latestOffset(
  startOffset: Offset,
  limit: ReadLimit): Offset

latestOffset is part of the SupportsAdmissionControl abstraction.

latestOffset is different whether it is executed in Trigger.AvailableNow mode or not (based on isTriggerAvailableNow flag).

latestOffset calculates next offset just once for Trigger.AvailableNow and the offsetForTriggerAvailableNow registry has no value assigned yet.

In other words, for isTriggerAvailableNow, latestOffset returns the value of the offsetForTriggerAvailableNow registry (that remains the same for every latestOffset).

For all the other triggers (when isTriggerAvailableNow is disabled), latestOffset calculates next offset.

calculateNextOffset(
  start: Offset): Offset

calculateNextOffset extractOffsetAndTimestamp from the given start Offset.

calculateNextOffset creates a RatePerMicroBatchStreamOffset for the end offset (that is rowsPerBatch rows long):

Increments the start offset by the rowsPerBatch for the end offset
Increments the start offset timestamp by the advanceMsPerBatch for the end offset timestamp