DeltaSQLConf — spark.databricks.delta Configuration Properties

The following table contains spark.databricks.delta.-prefixed configuration properties.

Table 1. spark.databricks.delta Configuration Properties
Name Description

import.batchSize.schemaInference

(internal) Number of files per batch for schema inference during import.

Default: 1000000

import.batchSize.statsCollection

(internal) Number of files per batch for stats collection during import.

Default: 50000

schema.autoMerge.enabled

Enables schema merging on appends and overwrites.

Equivalent DataFrame option: mergeSchema

Default: false

snapshotPartitions

(internal) Number of partitions to use for state reconstruction (when building a snapshot of a delta table).

Default: 50

timeTravel.resolveOnIdentifier.enabled

(internal) When enabled (true), patterns as @v123 in identifiers are considered as time travel nodes.

Default: true