Skip to content

RocksDBMetrics

RocksDBMetrics represents RocksDB stats of a load or a commit.

Creating Instance

RocksDBMetrics takes the following to be created:

  • numCommittedKeys
  • numUncommittedKeys
  • totalMemUsageBytes
  • writeBatchMemUsageBytes
  • totalSSTFilesBytes
  • nativeOpsHistograms
  • lastCommitLatencyMs
  • filesCopied
  • bytesCopied
  • filesReused
  • zipFileBytesUncompressed
  • nativeOpsMetrics (Map[String, Long])

RocksDBMetrics is created when:

  • RocksDB is requested for the metrics

lastCommitLatencyMs

lastCommitLatencyMs: Map[String, Long]

RocksDBMetrics is given durations (of each named commit phase of RocksDB committing a state version) when created.

Commit Phase Metric
checkpoint RocksDB: commit - checkpoint time
compact RocksDB: commit - compact time
fileSync RocksDB: commit - file sync to external storage time
flush RocksDB: commit - flush time
pauseBg RocksDB: commit - pause bg time
writeBatch RocksDB: commit - write batch time

SPARK-40807

SPARK-40807 "RocksDB: commit - pause bg time total" metric always 0

Used when RocksDBStateStore is requested for the metrics.

nativeOpsHistograms

nativeOpsHistograms: Map[String, RocksDBNativeHistogram]

RocksDBMetrics is given a lookup table of RocksDBNativeHistograms by name when created.

nativeOpsHistograms is created when RocksDB is requested for the metrics (based on the nativeOpsLatencyMicros with RocksDB's HistogramType and the nativeStats).

Native Op HistogramType
get DB_GET
put DB_WRITE
compaction COMPACTION_TIME

Used when RocksDBStateStore is requested for the metrics for the following: