CommitInfo

CommitInfo is an Action with the following:

  • Version

  • Timestamp

  • User ID

  • User Name

  • Name of the operation

  • Parameters of the operation

  • JobInfo

  • NotebookInfo

  • Cluster ID

  • Read Version

  • Isolation Level

  • isBlindAppend flag (to indicate whether a commit has blindly appended without caring about existing files)

  • Metrics of the operation

  • User metadata

CommitInfo is created (using apply utility) when:

  • OptimisticTransactionImpl is requested to commit

  • ConvertToDeltaCommand command is requested to streamWrite (when executed)

CommitInfo is used in OptimisticTransactionImpl and CommitStats.

CommitInfo is added (logged) to a Delta log only for spark.databricks.delta.commitInfo.enabled configuration enabled.

apply Utility

apply(
  time: Long,
  operation: String,
  operationParameters: Map[String, String],
  commandContext: Map[String, String],
  readVersion: Option[Long],
  isolationLevel: Option[String],
  isBlindAppend: Option[Boolean],
  operationMetrics: Option[Map[String, String]],
  userMetadata: Option[String]): CommitInfo

apply creates a CommitInfo (for the given arguments and based on the given commandContext for the user ID, user name, job, notebook, cluster).

commandContext is always empty, but could be customized using ConvertToDeltaCommandBase.

apply is used when:

  • OptimisticTransactionImpl is requested to commit

  • ConvertToDeltaCommand command is requested to streamWrite (when executed)