CommitInfo¶

CommitInfo is an Action defined by the following properties:

• Version (optional)
• Timestamp
• User ID (optional)
• User Name (optional)
• Operation
• Operation Parameters
• JobInfo (optional)
• NotebookInfo (optional)
• Cluster ID (optional)
• Read Version (optional)
• Isolation Level (optional)
• isBlindAppend flag (optional)
• Operation Metrics (optional)
• User metadata (optional)

CommitInfo is created (using apply and empty utilities) when:

• DeltaHistoryManager is requested for version and commit history (for DeltaTable.history operator and DESCRIBE HISTORY SQL command)
• OptimisticTransactionImpl is requested to commit (with spark.databricks.delta.commitInfo.enabled configuration property enabled)
• DeltaCommand is requested to commitLarge (for ConvertToDeltaCommand command and FileAlreadyExistsException was thrown)

CommitInfo is used as a part of OptimisticTransactionImpl and CommitStats.

Blind Append¶

CommitInfo is given isBlindAppend flag (when created) to indicate whether a commit has blindly appended data without caring about existing files.

isBlindAppend flag is used while checking for logical conflicts with concurrent updates (at commit).

isBlindAppend flag is always false when DeltaCommand is requested to commitLarge.

DeltaHistoryManager¶

CommitInfo can be looked up using DeltaHistoryManager for the following:

• DESCRIBE HISTORY SQL command
• DeltaTable.history operation

spark.databricks.delta.commitInfo.enabled¶

CommitInfo is added (logged) to a delta log only with spark.databricks.delta.commitInfo.enabled configuration property enabled.

Creating Empty CommitInfo¶

empty(
  version: Option[Long] = None): CommitInfo

empty...FIXME

empty is used when:

• DeltaHistoryManager is requested to getCommitInfo

Creating CommitInfo¶

apply(
  time: Long,
  operation: String,
  operationParameters: Map[String, String],
  commandContext: Map[String, String],
  readVersion: Option[Long],
  isolationLevel: Option[String],
  isBlindAppend: Option[Boolean],
  operationMetrics: Option[Map[String, String]],
  userMetadata: Option[String]): CommitInfo

apply creates a CommitInfo (for the given arguments and based on the given commandContext for the user ID, user name, job, notebook, cluster).

commandContext argument is always empty, but could be customized using ConvertToDeltaCommandBase.

apply is used when:

• OptimisticTransactionImpl is requested to commit (with spark.databricks.delta.commitInfo.enabled configuration property enabled)
• DeltaCommand is requested to commitLarge