DeltaConfigs¶
DeltaConfigs
is the configuration properties of a delta table.
ALTER TABLE SQL Command¶
Table properties can be set a value or unset using ALTER TABLE
SQL command:
ALTER TABLE <table_name> SET TBLPROPERTIES (<key>=<value>)
ALTER TABLE table1 UNSET TBLPROPERTIES [IF EXISTS] ('key1', 'key2', ...);
Configuration Properties¶
appendOnly¶
Whether a delta table is append-only (true
) or not (false
). When enabled, a table allows appends only and no updates or deletes.
Default: false
Used when:
DeltaLog
is requested to assertRemovable (that in turn usesDeltaErrors
utility to modifyAppendOnlyTableException)Protocol
utility is used to requiredMinimumProtocol
autoOptimize¶
Whether this delta table will automagically optimize the layout of files during writes.
Default: false
checkpointInterval¶
How often to checkpoint the state of a delta table
Default: 10
checkpointRetentionDuration¶
How long to keep checkpoint files around before deleting them
Default: interval 2 days
The most recent checkpoint is never deleted. It is acceptable to keep checkpoint files beyond this duration until the next calendar day.
checkpoint.writeStatsAsJson¶
Controls whether to write file statistics in the checkpoint in JSON format as the stats
column.
Default: true
checkpoint.writeStatsAsStruct¶
Controls whether to write file statistics in the checkpoint in the struct format in the stats_parsed
column and partition values as a struct as partitionValues_parsed
Default: undefined
(Option[Boolean]
)
compatibility.symlinkFormatManifest.enabled¶
Whether to register the GenerateSymlinkManifest post-commit hook while committing a transaction or not
Default: false
dataSkippingNumIndexedCols¶
The number of columns to collect stats on for data skipping. -1
means collecting stats for all columns.
Default: 32
deletedFileRetentionDuration¶
How long to keep logically deleted data files around before deleting them physically (to prevent failures in stale readers after compactions or partition overwrites)
Default: interval 1 week
enableExpiredLogCleanup¶
Whether to clean up expired log files and checkpoints
Default: true
enableFullRetentionRollback¶
Controls whether or not a delta table can be rolled back to any point within logRetentionDuration. When disabled, the table can be rolled back checkpointRetentionDuration only.
Default: true
logRetentionDuration¶
How long to keep obsolete logs around before deleting them. Delta can keep logs beyond the duration until the next calendar day to avoid constantly creating checkpoints.
Default: interval 30 days
(CalendarInterval
)
minReaderVersion¶
The protocol reader version
Default: 1
This property is not stored as a table property in the Metadata
action. It is stored as its own action. Having it modelled as a table property makes it easier to upgrade, and view the version.
minWriterVersion¶
The protocol reader version
Default: 3
This property is not stored as a table property in the Metadata
action. It is stored as its own action. Having it modelled as a table property makes it easier to upgrade, and view the version.
randomizeFilePrefixes¶
Whether to use a random prefix in a file path instead of partition information (may be required for very high volume S3 calls to better be partitioned across S3 servers)
Default: false
randomPrefixLength¶
The length of the random prefix in a file path for randomizeFilePrefixes
Default: 2
sampleRetentionDuration¶
How long to keep delta sample files around before deleting them
Default: interval 7 days
Building Configuration¶
buildConfig[T](
key: String,
defaultValue: String,
fromString: String => T,
validationFunction: T => Boolean,
helpMessage: String,
minimumProtocolVersion: Option[Protocol] = None): DeltaConfig[T]
buildConfig
creates a DeltaConfig for the given key
(with delta prefix added) and adds it to the entries internal registry.
buildConfig
is used to define all of the configuration properties in a type-safe way and (as a side effect) register them with the system-wide entries internal registry.
System-Wide Configuration Entries Registry¶
entries: HashMap[String, DeltaConfig[_]]
DeltaConfigs
utility (a Scala object) uses entries
internal registry of DeltaConfigs by their key.
New entries are added in buildConfig.
entries
is used when:
mergeGlobalConfigs Utility¶
mergeGlobalConfigs(
sqlConfs: SQLConf,
tableConf: Map[String, String],
protocol: Protocol): Map[String, String]
mergeGlobalConfigs
finds all spark.databricks.delta.properties.defaults-prefixed configuration properties among the entries.
mergeGlobalConfigs
is used when:
OptimisticTransactionImpl
is requested to withGlobalConfigDefaultsInitialSnapshot
is created
validateConfigurations Utility¶
validateConfigurations(
configurations: Map[String, String]): Map[String, String]
validateConfigurations
...FIXME
validateConfigurations
is used when:
DeltaCatalog
is requested to verifyTableAndSolidify and alterTable
normalizeConfigKeys Utility¶
normalizeConfigKeys(
propKeys: Seq[String]): Seq[String]
normalizeConfigKeys
...FIXME
normalizeConfigKeys
is used when:
- AlterTableUnsetPropertiesDeltaCommand is executed
spark.databricks.delta.properties.defaults Prefix¶
DeltaConfigs uses spark.databricks.delta.properties.defaults prefix for global configuration properties.