AdmissionLimits¶
AdmissionLimits is used by DeltaSource to control how much data should be processed by a single micro-batch.
Creating Instance¶
AdmissionLimits takes the following to be created:
- Maximum Number of Files (based on maxFilesPerTrigger option)
- Maximum Bytes (based on maxBytesPerTrigger option)
AdmissionLimits is created when:
DeltaSourceis requested to getChangesWithRateLimit, getStartingOffset, getDefaultReadLimit
Converting ReadLimit to AdmissionLimits¶
apply(
limit: ReadLimit): Option[AdmissionLimits]
apply creates an AdmissionLimits for the given ReadLimit (Spark Structured Streaming).
| ReadLimit | AdmissionLimits |
|---|---|
ReadAllAvailable | None |
ReadMaxFiles | Maximum Number of Files |
ReadMaxBytes | Maximum Bytes |
CompositeLimit | Maximum Number of Files and Maximum Bytes |
apply throws an UnsupportedOperationException for unknown ReadLimits:
Unknown ReadLimit: [limit]
apply is used when:
DeltaSourceis requested for the latest available offset
Admitting AddFile¶
admit(
add: Option[AddFile]): Boolean
admit...FIXME
admit is used when:
DeltaSourceis requested to getChangesWithRateLimit