DeltaSourceMetadataEvolutionSupport¶

DeltaSourceMetadataEvolutionSupport is an extension of the DeltaSourceBase abstraction to support DeltaSources.

Implementations¶

• DeltaSource

trackingMetadataChange Flag¶

trackingMetadataChange: Boolean

trackingMetadataChange is enabled (true) when all the following hold:

1. allowUnsafeStreamingReadOnColumnMappingSchemaChanges (of this SchemaReadOptions) is disabled
2. There is a PersistedMetadata (of this DeltaSourceMetadataTrackingLog)

trackingMetadataChange is used when:

• DeltaSource is requested to getFileChanges and getSnapshotAt
• DeltaSourceBase is requested to checkNonAdditiveSchemaChanges, checkReadIncompatibleSchemaChangeOnStreamStartOnce and getNextOffsetFromPreviousOffset
• DeltaSourceCDCSupport is requested to getFileChangesForCDC
• DeltaSourceMetadataEvolutionSupport is requested to getMetadataOrProtocolChangeIndexedFileIterator and updateMetadataTrackingLogAndFailTheStreamIfNeeded

initializeMetadataTrackingAndExitStream¶

initializeMetadataTrackingAndExitStream(
  batchStartVersion: Long,
  batchEndVersionOpt: Option[Long] = None,
  alwaysFailUponLogInitialized: Boolean = false): Unit

initializeMetadataTrackingAndExitStream...FIXME

initializeMetadataTrackingAndExitStream is used when:

• DeltaSource is requested to validateAndInitMetadataLogForPlannedBatchesDuringStreamStart
• DeltaSourceBase is requested to getStartingOffsetFromSpecificDeltaVersion

updateMetadataTrackingLogAndFailTheStreamIfNeeded¶

updateMetadataTrackingLogAndFailTheStreamIfNeeded(
  end: Offset): Unit
updateMetadataTrackingLogAndFailTheStreamIfNeeded(
  changedMetadataOpt: Option[Metadata],
  changedProtocolOpt: Option[Protocol],
  version: Long,
  replace: Boolean = false): Unit

updateMetadataTrackingLogAndFailTheStreamIfNeeded...FIXME

updateMetadataTrackingLogAndFailTheStreamIfNeeded is used when:

• DeltaSource is requested to getSnapshotAt and commit