ShuffleWriteMetricsReporter¶

ShuffleWriteMetricsReporter is an abstraction of shuffle write metrics reporters.

Contract¶

decBytesWritten¶

decBytesWritten(
  v: Long): Unit

decRecordsWritten¶

decRecordsWritten(
  v: Long): Unit

incBytesWritten¶

incBytesWritten(
  v: Long): Unit

incRecordsWritten¶

incRecordsWritten(
  v: Long): Unit

See ShuffleWriteMetrics

Used when:

• ShufflePartitionPairsWriter is requested to recordWritten
• ShuffleExternalSorter is requested to writeSortedFile
• DiskBlockObjectWriter is requested to record bytes written

incWriteTime¶

incWriteTime(
  v: Long): Unit

Used when:

• BypassMergeSortShuffleWriter is requested to write partition records and writePartitionedData
• UnsafeShuffleWriter is requested to mergeSpillsWithTransferTo
• DiskBlockObjectWriter is requested to commitAndGet
• TimeTrackingOutputStream is requested to write, flush, and close

Implementations¶

• ShuffleWriteMetrics
• SQLShuffleWriteMetricsReporter (Spark SQL)