Skip to content


KafkaDataWriter is a DataWriter.


KafkaDataWriter is a KafkaRowWriter (for the input schema and topic).

Creating Instance

KafkaDataWriter takes the following to be created:

  • Optional Topic Name (target topic)
  • Kafka Producer Parameters
  • Input Schema (Attributes)

KafkaDataWriter is created when:

Cached KafkaProducer

producer: Option[CachedKafkaProducer] = None

KafkaDataWriter defines producer internal registry for a CachedKafkaProducer:

  • producer is undefined when KafkaDataWriter is created
  • CachedKafkaProducer is created (aquired) when writing out a row
  • producer is cleared (dereferenced) in close

Once defined, KafkaDataWriter uses the KafkaProducer to send a row (when writing out a row).

KafkaDataWriter requests the KafkaProducer to flush out rows in commit.

FIXME: Why is InternalKafkaProducerPool required?