KafkaSourceOffset¶

KafkaSourceOffset is an Offset in Kafka Data Source.

Creating Instance¶

KafkaSourceOffset takes the following to be created:

• Offsets by Partitions (Map[TopicPartition, Long])

KafkaSourceOffset is created when:

• KafkaContinuousStream is requested for the initial offset, deserializeOffset, mergeOffsets
• KafkaMicroBatchStream is requested for the initial offset, reportLatestOffset, latestOffset, deserializeOffset, getOrCreateInitialPartitionOffsets
• KafkaOffsetReaderAdmin is requested to fetchSpecificOffsets0
• KafkaOffsetReaderConsumer is requested to fetchSpecificOffsets0
• KafkaSource is requested for the initialPartitionOffsets, reportLatestOffset, latestOffset
• KafkaSourceOffset utility is used to create a KafkaSourceOffset

Creating KafkaSourceOffset Instance¶

apply(
  offsetTuples: (String, Int, Long)*): KafkaSourceOffset
apply(
  offset: SerializedOffset): KafkaSourceOffset
apply(
  offset: Offset): KafkaSourceOffset

apply creates a KafkaSourceOffset.

apply is used when:

• KafkaMicroBatchStream is requested for the performance metrics
• KafkaSourceInitialOffsetWriter is requested to deserialize a KafkaSourceOffset
• KafkaSourceOffset is requested for the partition offsets

Partition Offsets¶

getPartitionOffsets(
  offset: Offset): Map[TopicPartition, Long]

getPartitionOffsets requests the given KafkaSourceOffset or SerializedOffset (from the given Offset) for the partitionToOffsets.

getPartitionOffsets is used when:

• KafkaSource is requested to getBatch