Skip to content

CommitInfo

CommitInfo is an Action defined by the following properties:

CommitInfo is created (using apply and empty utilities) when:

CommitInfo is used as a part of OptimisticTransactionImpl and CommitStats.

engineInfo

CommitInfo can be given extra engineInfo identifier (when created) for the engine that made the commit.

This engineInfo is by default getEngineInfo.

getEngineInfo

getEngineInfo: Option[String]

getEngineInfo is the following text:

Apache-Spark/[SPARK_VERSION] Delta-Lake/[VERSION]

Blind Append

CommitInfo is given isBlindAppend flag (when created) to indicate whether a commit has blindly appended data without caring about existing files.

isBlindAppend flag is used while checking for logical conflicts with concurrent updates (at commit).

isBlindAppend flag is always false when DeltaCommand is requested to commitLarge.

DeltaHistoryManager

CommitInfo can be looked up using DeltaHistoryManager for the following:

spark.databricks.delta.commitInfo.enabled

CommitInfo is added (logged) to a delta log only with spark.databricks.delta.commitInfo.enabled configuration property enabled.

Creating Empty CommitInfo

empty(
  version: Option[Long] = None): CommitInfo

empty...FIXME

empty is used when:

Creating CommitInfo

apply(
  time: Long,
  operation: String,
  operationParameters: Map[String, String],
  commandContext: Map[String, String],
  readVersion: Option[Long],
  isolationLevel: Option[String],
  isBlindAppend: Option[Boolean],
  operationMetrics: Option[Map[String, String]],
  userMetadata: Option[String],
  tags: Option[Map[String, String]],
  txnId: Option[String]): CommitInfo

apply creates a CommitInfo (for the given arguments and based on the given commandContext for the user ID, user name, job, notebook, cluster).

commandContext argument is always empty, but could be customized using ConvertToDeltaCommandBase.

apply is used when: