Skip to content

DeltaOptions

[[DeltaWriteOptionsImpl]][[DeltaWriteOptions]][[DeltaReadOptions]] DeltaOptions (aka DeltaWriteOptionsImpl, DeltaWriteOptions) is the options for the <>.

The <> can be defined using option method of DataFrameReader, DataFrameWriter, DataStreamReader, and DataStreamWriter.

DeltaOptions is used to create <> command, <>, and <>.

DeltaOptions can be verified.

Options

=== [[checkpointLocation]] checkpointLocation

=== [[DATA_CHANGE_OPTION]][[dataChange]] dataChange

=== [[EXCLUDE_REGEX_OPTION]][[excludeRegex]] excludeRegex

=== [[IGNORE_CHANGES_OPTION]][[ignoreChanges]] ignoreChanges

=== [[IGNORE_DELETES_OPTION]][[ignoreDeletes]] ignoreDeletes

=== [[IGNORE_FILE_DELETION_OPTION]][[ignoreFileDeletion]] ignoreFileDeletion

=== [[MAX_BYTES_PER_TRIGGER_OPTION]][[maxBytesPerTrigger]] maxBytesPerTrigger

=== [[MAX_FILES_PER_TRIGGER_OPTION]][[maxFilesPerTrigger]][[MAX_FILES_PER_TRIGGER_OPTION_DEFAULT]] maxFilesPerTrigger

Maximum number of files (<>) that <> will <> (read) in a streaming micro-batch (trigger)

Default: 1000

Must be at least 1

=== [[MERGE_SCHEMA_OPTION]][[mergeSchema]][[canMergeSchema]] mergeSchema

Enables schema migration (e.g. allows automatic schema merging during a write operation for <> and <>)

Equivalent SQL Session configuration: <>

=== [[OPTIMIZE_WRITE_OPTION]][[optimizeWrite]] optimizeWrite

=== [[OVERWRITE_SCHEMA_OPTION]][[overwriteSchema]] overwriteSchema

=== [[path]] path

(required) Directory on a Hadoop DFS-compliant file system with an optional <> identifier.

Default: (undefined)

Can also be specified using load method of DataFrameReader and DataStreamReader.

=== [[queryName]] queryName

=== [[REPLACE_WHERE_OPTION]][[replaceWhere]] replaceWhere

=== [[timestampAsOf]] timestampAsOf

<> using a timestamp of a table

Mutually exclusive with <> option and the time travel identifier of the <> option.

=== [[USER_METADATA_OPTION]][[userMetadata]] userMetadata

Defines a CommitInfo.md#userMetadata[user-defined commit metadata]

Take precedence over DeltaSQLConf.md#commitInfo.userMetadata[spark.databricks.delta.commitInfo.userMetadata]

Available by inspecting CommitInfo.md[]s using delta-sql.md#DESCRIBE-HISTORY[DESCRIBE HISTORY] or DeltaTable.md#history[DeltaTable.history].

=== [[versionAsOf]] versionAsOf

<> using a version of a table

Mutually exclusive with <> option and the time travel identifier of the <> option.

Used exclusively when DeltaDataSource is requested for a <>

== [[creating-instance]] Creating Instance

DeltaOptions takes the following to be created:

  • Options (Map[String, String] or CaseInsensitiveMap[String])
  • [[sqlConf]] SQLConf

DeltaOptions is created when:

  • DeltaLog is requested to <> (for <> as a <> and a <>)

  • DeltaDataSource is requested to <> (to create a <> for Structured Streaming), <> (to create a <> for Structured Streaming), and <>

Verifying Options

verifyOptions(
  options: CaseInsensitiveMap[String]): Unit

verifyOptions...FIXME


Last update: 2020-10-13