DeltaReadOptions¶

DeltaReadOptions is an extension of the DeltaOptionParser abstraction with the values of the read options of DeltaOptions.

excludeRegex¶

excludeRegex: Option[Regex]

excludeRegex uses the options for the value of excludeRegex option and converts it to a scala.util.matching.Regex.

excludeRegex is used when:

• DeltaSource is requested for the excludeRegex

failOnDataLoss¶

failOnDataLoss: Boolean

failOnDataLoss uses the options for the value of failOnDataLoss option.

failOnDataLoss is true by default.

failOnDataLoss is used when:

• DeltaSource is requested to getFileChanges
• DeltaSourceCDCSupport is requested to getFileChangesForCDC

ignoreChanges¶

ignoreChanges: Boolean

ignoreChanges...FIXME

ignoreChanges is used when:

• FIXME

ignoreDeletes¶

ignoreDeletes: Boolean

ignoreDeletes...FIXME

ignoreDeletes is used when:

• FIXME

ignoreFileDeletion¶

ignoreFileDeletion: Boolean

ignoreFileDeletion...FIXME

ignoreFileDeletion is used when:

• FIXME

maxBytesPerTrigger¶

maxBytesPerTrigger: Option[Long]

maxBytesPerTrigger...FIXME

maxBytesPerTrigger is used when:

• FIXME

maxFilesPerTrigger¶

maxFilesPerTrigger: Option[Int]

maxFilesPerTrigger...FIXME

maxFilesPerTrigger is used when:

• FIXME

readChangeFeed¶

readChangeFeed: Boolean

readChangeFeed uses the options for the boolean value of readChangeFeed option (if available or falls back to the legacy readChangeData).

readChangeFeed is used when DeltaSourceBase is requested for the following:

• checkReadIncompatibleSchemaChanges
• getFileChangesAndCreateDataFrame (for the DeltaSource to get a streaming micro-batch dataframe)
• getFileChangesWithRateLimit (for DeltaSource to determine the latest offset)
• The read schema

startingTimestamp¶

startingTimestamp: Option[String]

startingTimestamp...FIXME

startingTimestamp is used when:

• FIXME

startingVersion¶

startingVersion: Option[DeltaStartingVersion]

startingVersion...FIXME

startingVersion is used when:

• FIXME