DeltaSourceOffset¶

DeltaSourceOffset is a streaming Offset (Spark Structured Streaming) for DeltaSource.

Creating Instance¶

DeltaSourceOffset takes the following to be created:

• Source Version
• Reservoir ID (aka Table ID)
• Reservoir Version
• Index
• isStartingVersion flag

DeltaSourceOffset is created (using apply utility) when:

• DeltaSource is requested for the starting and latest offsets

isStartingVersion Flag¶

DeltaSourceOffset is given isStartingVersion flag when created to denote a query that is starting rather than processing changes.

isStartingVersion flag is false when:

• DeltaSourceBase is requested to buildOffsetFromIndexedFile with the last IndexedFile for a given version

isStartingVersion flag is copied over (continued) as long as the versions are the same when buildOffsetFromIndexedFile

Creating DeltaSourceOffset¶

apply(
  reservoirId: String,
  offset: Offset): DeltaSourceOffset
apply(
  reservoirId: String,
  reservoirVersion: Long,
  index: Long,
  isStartingVersion: Boolean): DeltaSourceOffset

apply creates a DeltaSourceOffset (for the version and the given arguments) or converts a SerializedOffset to a DeltaSourceOffset.

apply is used when:

• DeltaSource is requested for the starting and latest offsets

validateSourceVersion¶

validateSourceVersion(
  json: String): Unit

validateSourceVersion...FIXME

Source Version¶

DeltaSourceOffset uses 1 for the version (and does not allow changing it).

The version is used when:

• DeltaSourceOffset.apply and validateSourceVersion utilities are used