- Auto Compaction
- Change Data Feed
- CHECK Constraints
- Column Invariants
- Column Mapping
- Column Statistics
- Data Skipping
- Delta SQL
- Developer API
- Generated Columns
- Spark SQL integration with support for batch and streaming queries
- Table Constraints
- Time Travel
- others (listed in the menu on the left)
Delta Lake allows you to store data on blob stores like HDFS, S3, Azure Data Lake, GCS, query from many processing engines including Apache Spark, Trino, Apache Hive, Apache Flink, and provides APIs for SQL, Scala, Java, Python, Rust (to name a few).
Delta tables can be registered in a table catalog. Delta Lake creates a transaction log at the root directory of a table, and the catalog contains no information but the table format and the location of the table. All table properties, schema and partitioning information live in the transaction log to avoid a "split brain" situation (Wikipedia).
Delta Lake 3.1.0 supports Apache Spark 3.5.0 (cf. build.sbt).
Delta Lake uses OptimisticTransaction for transactional writes. A commit is successful when the transaction can write the actions to a delta file (in the transactional log). In case the delta file for the commit version already exists, the transaction is retried.
More importantly, multiple queries can write to the same delta table simultaneously (at exactly the same time).
TransactionalWrite is an interface for writing out data to a delta table.
The following commands and interfaces can transactionally write new data files out to a data directory of a delta table:
Delta Lake provides the following Developer APIs for developers to interact with (and even extend) Delta Lake using a supported programming language:
Delta Lake supports batch and streaming queries (Spark SQL and Structured Streaming, respectively) using delta format.
In order to fine tune queries over data in Delta Lake use options.
Structured queries can write (transactionally) to a delta table using the following interfaces:
- WriteIntoDelta command for batch queries (Spark SQL)
- DeltaSink for streaming queries (Spark Structured Streaming)
Delta Lake supports reading and writing in batch queries:
Delta Lake supports reading and writing in streaming queries:
Delta Tables in Logical Query Plans¶
Put simply, delta tables are
HadoopFsRelation with TahoeFileIndex in logical query plans.
Concurrent Blind Append Transactions¶
Blind append transactions are marked in the commit info to distinguish them from read-modify-appends (deletes, merges or updates) and assume no conflict between concurrent transactions.
Blind Append Transactions allow for concurrent updates.
Exception Public API¶
Delta Lake introduces exceptions due to conflicts between concurrent operations as a public API.
- What's New in Delta Lake 2.3.0 by Will Girten