DeltaConfigs — Table Property

DeltaConfigs (of type T) represents a named property of a delta table with values (of type T) in a validated range.

DeltaConfigs is persisted to the Metadata.

DeltaConfigs table property can be assigned a value using ALTER TABLE SQL command:

ALTER TABLE <table_name> SET TBLPROPERTIES (<key>=<value>)
Table 1. DeltaConfigs
Key Description

autoOptimize

Whether this delta table will automagically optimize the layout of files during writes.

Default: false

checkpointInterval

How often to checkpoint the state of a delta table

Default: 10

checkpointRetentionDuration

How long to keep checkpoint files around before deleting them

Default: interval 2 days

The most recent checkpoint is never deleted. It is acceptable to keep checkpoint files beyond this duration until the next calendar day.

dataSkippingNumIndexedCols

The number of columns to collect stats on for data skipping. -1 means collecting stats for all columns.

Default: 32

enableExpiredLogCleanup

Whether to clean up expired log files and checkpoints

Default: true

enableFullRetentionRollback

When enabled (default), a delta table can be rolled back to any point within logRetentionDuration. When disabled, the table can be rolled back checkpointRetentionDuration only.

Default: true

appendOnly

Whether a delta table is append-only (true) or not (false). When enabled, a table allows appends only and no updates or deletes.

Default: false

logRetentionDuration

How long to keep obsolete logs around before deleting them. Delta can keep logs beyond the duration until the next calendar day to avoid constantly creating checkpoints.

Default: interval 30 days

randomizeFilePrefixes

Whether to use a random prefix in a file path instead of partition information (may be required for very high volume S3 calls to better be partitioned across S3 servers)

Default: false

randomPrefixLength

The length of the random prefix in a file path for randomizeFilePrefixes

Default: 2

sampleRetentionDuration

How long to keep delta sample files around before deleting them

Default: interval 7 days

deletedFileRetentionDuration

How long to keep logically deleted data files around before deleting them physically (to prevent failures in stale readers after compactions or partition overwrites)

Default: interval 1 week

Creating DeltaConfigs Instance

DeltaConfigs takes the following to be created:

  • Key

  • Default value

  • Conversion function (from text representation of the DeltaConfigs to the type, i.e. String ⇒ T)

  • Validation function (that guards from incorrect values, i.e. T ⇒ Boolean)

  • Help message

  • (optional) Minimum version of protocol supported

DeltaConfigs initializes the internal properties.

Reading Configuration Property From Metadata — fromMetaData Method

fromMetaData(metadata: Metadata): T

fromMetaData looks up the key of the DeltaConfigs configuration in the configuration of the given Metadata. If not found, fromMetaData takes the default value.

In the end, fromMetaData converts the text representation to the proper type using fromString conversion function.

fromMetaData is used when:

mergeGlobalConfigs Utility

mergeGlobalConfigs(
  sqlConfs: SQLConf,
  tableConf: Map[String, String],
  protocol: Protocol): Map[String, String]

mergeGlobalConfigs…​FIXME

mergeGlobalConfigs is used when…​FIXME