BasicWriteTaskStatsTracker is a WriteTaskStatsTracker.
BasicWriteTaskStatsTracker takes the following to be created:
BasicWriteTaskStatsTracker is created when:
BasicWriteJobStatsTrackeris requested for a new WriteTaskStatsTracker instance
submittedFiles registry of the file paths added (written out to).
A file path is removed when
BasicWriteTaskStatsTracker is requested to closeFile.
All the file paths are removed in getFinalStats.
updateFileStats( filePath: String): Unit
updateFileStats gets the size of the given
Processing New File Notification¶
newFile( filePath: String): Unit
newFile is part of the WriteTaskStatsTracker abstraction.
getFinalStats( taskCommitTime: Long): WriteTaskStats
getFinalStats sets the output metrics (of the current Spark task) as follows:
Expected [numSubmittedFiles] files, but only saw $numFiles. This could be due to the output format not writing empty files, or files being not immediately visible in the filesystem.
getFinalStats adds the given
taskCommitTime to the taskCommitTimeMetric if defined.
getFinalStats is part of the WriteTaskStatsTracker abstraction.
ALL logging level for
org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker logger to see what happens inside.
Add the following line to
Refer to Logging.