Skip to content

DataWriter

DataWriter is an abstraction of data writers (of data of type T).

Contract

Aborting Write

void abort()

Used when:

  • FileFormatWriter utility is used to executeTask
  • DataWritingSparkTask utility is used to process a partition
  • ContinuousWriteRDD (Spark Structured Streaming) is requested to compute a partition

Committing Successful Write

WriterCommitMessage commit()

Used when:

  • DataWritingSparkTask utility is used to process a partition
  • ContinuousWriteRDD (Spark Structured Streaming) is requested to compute a partition

currentMetricsValues

CustomTaskMetric[] currentMetricsValues()

Used when:

  • FileFormatWriter utility is used to executeTask
  • DataWritingSparkTask utility is used to process a partition
  • ContinuousWriteRDD (Spark Structured Streaming) is requested to compute a partition

Writing Out Record

void write(
  T record)

Used when:

Implementations

  • FileFormatDataWriter
  • ForeachDataWriter (Spark Structured Streaming)
  • KafkaDataWriter
  • MemoryDataWriter (Spark Structured Streaming)
  • NoopWriter
  • PackedRowDataWriter (Spark Structured Streaming)