KafkaDataWriter is a DataWriter.
KafkaDataWriter takes the following to be created:
- Optional Topic Name (target topic)
- Kafka Producer Parameters
- Input Schema (Attributes)
KafkaDataWriter is created when:
KafkaBatchWriterFactoryis requested for a DataWriter
KafkaStreamWriterFactory(Spark Structured Streaming) is requested for a
producer: Option[CachedKafkaProducer] = None
producer internal registry for a
produceris undefined when
CachedKafkaProduceris created (aquired) when writing out a row
produceris cleared (dereferenced) in close
KafkaDataWriter requests the
KafkaProducer to flush out rows in commit.
FIXME: Why is InternalKafkaProducerPool required?