KafkaSourceInitialOffsetWriter¶

KafkaSourceInitialOffsetWriter is a HDFSMetadataLog of KafkaSourceOffsets.

Creating Instance¶

KafkaSourceInitialOffsetWriter takes the following to be created:

• SparkSession (Spark SQL)
• Path of the metadata log directory

KafkaSourceInitialOffsetWriter is created when:

• KafkaMicroBatchStream is requested to getOrCreateInitialPartitionOffsets
• KafkaSource is requested to getOrCreateInitialPartitionOffsets

deserialize¶

deserialize(
  in: InputStream): KafkaSourceOffset

deserialize is part of the HDFSMetadataLog abstraction.

deserialize creates a KafkaSourceOffset (from a SerializedOffset).