MetadataLog¶

MetadataLog is an abstraction of metadata logs that can add, get, getLatest and purge metadata (of type T).

Contract¶

Storing Metadata of Streaming Batch¶

add(
  batchId: Long,
  metadata: T): Boolean

Stores (adds) metadata of a streaming batch

Used when:

• KafkaSource is requested for the initialPartitionOffsets

• CompactibleFileStreamLog is requested for the store metadata of a streaming batch and to compact

• FileStreamSource is requested to fetchMaxOffset

• FileStreamSourceLog is requested to store (add) metadata of a streaming batch

• ManifestFileCommitProtocol is requested to commitJob

• MicroBatchExecution stream execution engine is requested to <> and <>

• ContinuousExecution stream execution engine is requested to <> and <>

get¶

get(
  batchId: Long): Option[T]
get(
  startId: Option[Long],
  endId: Option[Long]): Array[(Long, T)]

Looks up (gets) metadata of one or more streaming batches

Used when...FIXME

getLatest¶

getLatest(): Option[(Long, T)]

Looks up the latest-committed metadata (if available)

Used when...FIXME

purge¶

purge(
  thresholdBatchId: Long): Unit

Purging (removing) metadata older than the given threshold

Used when...FIXME

Implementations¶

• HDFSMetadataLog