FileFormatDataWriter is an extension of the DataWriter abstraction for data writers (of InternalRows).
Writing Record Out¶
write( record: InternalRow): Unit
FileFormatDataWriteris requested to writeWithMetrics
FileFormatDataWriter takes the following to be created:
- Custom SQLMetrics by name (
FileFormatDataWriter is an abstract class and cannot be created directly. It is created indirectly for the concrete FileFormatDataWriters.
writeWithMetrics( record: InternalRow, count: Long): Unit
writeWithMetrics updates the CustomTaskMetrics with the customMetrics and writes out the given InternalRow.
writeWithMetrics is used when:
FileFormatDataWriteris requested to write out (a collection of) records
Writing Out (Collection of) Records¶
writeWithIterator( iterator: Iterator[InternalRow]): Unit
writeWithIterator is used when:
FileFormatWriterutility is used to write data out in a single Spark task
Committing Successful Write¶
commit requests the FileCommitProtocol to
commitTask (that gives a
commit creates a new
ExecutedWriteSummary with the updatedPartitions and the WriteTaskStats of the WriteTaskStatsTrackers.
In the end,
commit creates a
WriteTaskResult (for the
TaskCommitMessage and the
commit is part of the DataWriter abstraction.