Skip to content

Storage

Delta Lake can now automatically load the correct LogStore needed for common storage systems hosting the Delta table being read or written to.

LogStoreProvider uses DelegatingLogStore unless spark.delta.logStore.class configuration property is defined.

The scheme of the Delta table path is used to dynamically load the necessary LogStore implementation. This also allows the same application to simultaneously read and write to Delta tables on different cloud storage systems.

DelegatingLogStore allows for custom LogStores per URI scheme before using the default LogStores.

Schemes Default LogStore
s3, s3a, s3n S3SingleDriverLogStore
abfs, abfss, adl, wasb, wasbs AzureLogStore

DelegatingLogStore uses HDFSLogStore as the default LogStore.