KafkaScan is a Scan (a logical scan over data in Apache Kafka).
KafkaScan takes the following to be created:
- Case-Insensitive Options
KafkaScan is created when:
KafkaTableis requested for a ScanBuilder
readSchema is part of the Scan abstraction.
readSchema builds the read schema (possibly with records headers based on includeHeaders option).
Supported Custom Metrics¶
supportedCustomMetrics is part of the Scan abstraction.
supportedCustomMetrics gives the following CustomMetrics:
toMicroBatchStream( checkpointLocation: String): MicroBatchStream
toMicroBatchStream is part of the Scan abstraction.
toMicroBatchStream determines KafkaOffsetRangeLimit based on the following options (with
LatestOffsetRangeLimit as the default):
toMicroBatchStream builds a KafkaOffsetReader for the following:
| ||Kafka Configuration Properties for Driver|
| ||streamingUniqueGroupId with |
In the end,
toMicroBatchStream creates a
KafkaMicroBatchStream (Spark Structured Streaming) with the following: