SupportsAdmissionControl¶

SupportsAdmissionControl is an extension of the SparkDataStream abstraction for data streams to control the rate of data ingested (read) in Micro-Batch Stream Processing.

Contract¶

ReadLimit getDefaultReadLimit()

Note

MicroBatchExecution stream execution engine uses getDefaultReadLimit for SparkDataStreams in streaming queries with the following Triggers:

Default: ReadAllAvailable (and is expected to be overriden by the implementations, if needed)

See:

Used when:

MicroBatchExecution stream execution engine is requested for the analyzed logical plan (and initializes uniqueSources registry based on TriggerExecutor)

Offset latestOffset(
  Offset startOffset,
  ReadLimit limit)

The most recent Offset available given a ReadLimit. null to "announce" no data to process.

MicroBatchExecution and latestOffset Phase

MicroBatchExecution stream execution engine uses latestOffset execution phase to track the duration to request a SparkDataStream (indirectly via AvailableNowDataStreamWrapper) or SupportsAdmissionControl for latestOffset and reportLatestOffset

Used when:

AvailableNowDataStreamWrapper is requested to prepareForTriggerAvailableNow
MicroBatchExecution stream execution engine is requested to construct the next micro-batch

Offset reportLatestOffset()

The most recent Offset available (regardless of ReadLimit)

Default: null (no offset to process)

MicroBatchExecution and latestOffset Phase

MicroBatchExecution stream execution engine uses latestOffset execution phase to track the duration to request a SparkDataStream (indirectly via AvailableNowDataStreamWrapper) or SupportsAdmissionControl for latestOffset and reportLatestOffset

Used when:

AvailableNowDataStreamWrapper is requested to reportLatestOffset
MicroBatchExecution stream execution engine is requested to construct the next micro-batch (for AvailableNowDataStreamWrapper and SupportsAdmissionControl data streams)