DeltaOptions¶

DeltaOptions is a type-safe abstraction of write and read options.

DeltaOptions is used to create WriteIntoDelta command, DeltaSink, and DeltaSource.

DeltaOptions is a Serializable (Java) (so it can be used in Spark tasks).

Creating Instance¶

DeltaOptions takes the following to be created:

When created, DeltaOptions verifies the input options.

DeltaOptions is created when:

DeltaLog is requested for a relation (for DeltaDataSource as a CreatableRelationProvider and a RelationProvider)
DeltaCatalog is requested to createDeltaTable
WriteIntoDeltaBuilder is requested to buildForV1Write
CreateDeltaTableCommand is executed
DeltaDataSource is requested for a streaming source (to create a DeltaSource for Structured Streaming), a streaming sink (to create a DeltaSink for Structured Streaming), and for an insertable HadoopFsRelation

verifyOptions(
  options: CaseInsensitiveMap[String]): Unit

verifyOptions finds invalid options among the input options.

Note

In the open-source version verifyOptions does nothing. The underlying objects (recordDeltaEvent and the others) are no-ops.

verifyOptions is used when:

DeltaOptions is created
DeltaDataSource is requested for a relation (for loading data in batch queries)