KafkaDataWriter¶

KafkaDataWriter is a DataWriter.

KafkaRowWriter¶

KafkaDataWriter is a KafkaRowWriter (for the input schema and topic).

Creating Instance¶

KafkaDataWriter takes the following to be created:

• Optional Topic Name (target topic)
• Kafka Producer Parameters
• Input Schema (Attributes)

KafkaDataWriter is created when:

• KafkaBatchWriterFactory is requested for a DataWriter
• KafkaStreamWriterFactory (Spark Structured Streaming) is requested for a DataWriter

Cached KafkaProducer¶

producer: Option[CachedKafkaProducer] = None

KafkaDataWriter defines producer internal registry for a CachedKafkaProducer:

• producer is undefined when KafkaDataWriter is created
• CachedKafkaProducer is created (aquired) when writing out a row
• producer is cleared (dereferenced) in close

Once defined, KafkaDataWriter uses the KafkaProducer to send a row (when writing out a row).

KafkaDataWriter requests the KafkaProducer to flush out rows in commit.