OffsetSeqMetadata — Metadata of Streaming Batch¶

OffsetSeqMetadata holds the metadata for the current streaming batch:

• [[batchWatermarkMs]] Event-time watermark threshold

• [[batchTimestampMs]] Batch timestamp (in millis)

• [[conf]] Streaming configuration with spark.sql.shuffle.partitions and spark.sql.streaming.stateStore.providerClass configuration properties

OffsetSeqMetadata is used mainly when IncrementalExecution is created.

[[relevantSQLConfs]] OffsetSeqMetadata considers some configuration properties as relevantSQLConfs:

• SHUFFLE_PARTITIONS
• STATE_STORE_PROVIDER_CLASS
• STREAMING_MULTIPLE_WATERMARK_POLICY
• FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION
• STREAMING_AGGREGATION_STATE_FORMAT_VERSION

relevantSQLConfs are used when OffsetSeqMetadata is created and is requested to setSessionConf.

Creating OffsetSeqMetadata¶

apply(
  batchWatermarkMs: Long,
  batchTimestampMs: Long,
  sessionConf: RuntimeConfig): OffsetSeqMetadata

apply...FIXME

apply is used when...FIXME

=== [[setSessionConf]] setSessionConf Method

[source, scala]¶

setSessionConf(metadata: OffsetSeqMetadata, sessionConf: RuntimeConfig): Unit¶

setSessionConf...FIXME

NOTE: setSessionConf is used when...FIXME