DeltaOptions¶
DeltaOptions
is a type-safe abstraction of the supported write and read options.
DeltaOptions
is used to create WriteIntoDelta command, DeltaSink, and DeltaSource.
Creating Instance¶
DeltaOptions
takes the following to be created:
- Case-Insensitive Options
-
SQLConf
(Spark SQL)
When created, DeltaOptions
verifies the options.
DeltaOptions
is created when:
DeltaLog
is requested for a relation (for DeltaDataSource as a CreatableRelationProvider and a RelationProvider)DeltaCatalog
is requested to createDeltaTableWriteIntoDeltaBuilder
is requested to buildForV1Write- CreateDeltaTableCommand is executed
DeltaDataSource
is requested for a streaming source (to create a DeltaSource for Structured Streaming), a streaming sink (to create a DeltaSink for Structured Streaming), and for an insertable HadoopFsRelation
Verifying Options¶
verifyOptions(
options: CaseInsensitiveMap[String]): Unit
verifyOptions
finds invalid options among the input options
.
Note
In the open-source version verifyOptions
does really nothing. The underlying objects (recordDeltaEvent
and the others) are no-ops.
verifyOptions
is used when:
DeltaOptions
is createdDeltaDataSource
is requested for a relation (for loading data in batch queries)
Serializable¶
DeltaOptions
is a Serializable
(Java) (so it can be used in Spark tasks).