Skip to content

Metadata

Metadata is an Action to update the metadata of a delta table (indirectly via the Snapshot).

Use DescribeDeltaDetailCommand to review the metadata of a delta table.

Creating Instance

Metadata takes the following to be created:

  • Id (default: random UUID)
  • Name (default: null)
  • Description (default: null)
  • Format (default: empty)
  • Schema (default: null)
  • Partition Columns (default: Nil)
  • Table Configuration (default: Map.empty)
  • Created Time (default: current time)

Metadata is created when:

Updating Metadata

Metadata can be updated in a transaction once only (and only when created for an uninitialized table, when readVersion is -1).

txn.metadata

Demo

val path = "/tmp/delta/users"

import org.apache.spark.sql.delta.DeltaLog
val deltaLog = DeltaLog.forTable(spark, path)

import org.apache.spark.sql.delta.actions.Metadata
assert(deltaLog.snapshot.metadata.isInstanceOf[Metadata])

deltaLog.snapshot.metadata.id

Table ID

Metadata uses a Table ID (aka reservoirId) to uniquely identify a delta table and is never going to change through the history of the table.

Table ID is given Metadata is created or defaults to a random UUID (Java).

Note

When I asked the question tableId and reservoirId - Why two different names for metadata ID? on delta-users mailing list, Tathagata Das wrote:

Any reference to "reservoir" is just legacy code. In the early days of this project, the project was called "Tahoe" and each table is called a "reservoir" (Tahoe is one of the 2nd deepest lake in US, and is a very large reservoir of water ;) ). So you may still find those two terms all around the codebase.

In some cases, like DeltaSourceOffset, the term reservoirId is in the json that is written to the streaming checkpoint directory. So we cannot change that for backward compatibility.


Last update: 2021-06-02
Back to top