CommitInfo¶
CommitInfo
is an Action defined by the following properties:
- Version (optional)
- Timestamp
- User ID (optional)
- User Name (optional)
- Operation
- Operation Parameters
- JobInfo (optional)
- NotebookInfo (optional)
- Cluster ID (optional)
- Read Version (optional)
- Isolation Level (optional)
- isBlindAppend flag (optional)
- Operation Metrics (optional)
- User metadata (optional)
CommitInfo
is created (using apply and empty utilities) when:
DeltaHistoryManager
is requested for version and commit history (for DeltaTable.history operator and DESCRIBE HISTORY SQL command)OptimisticTransactionImpl
is requested to commit (with spark.databricks.delta.commitInfo.enabled configuration property enabled)DeltaCommand
is requested to commitLarge (for ConvertToDeltaCommand command)
CommitInfo
is used as a part of OptimisticTransactionImpl and CommitStats
.
isBlindAppend flag¶
CommitInfo
is given isBlindAppend
flag when created.
isBlindAppend
flag indicates whether a commit has blindly appended data without caring about existing files.
DeltaHistoryManager¶
CommitInfo
can be looked up using DeltaHistoryManager for the following:
- DESCRIBE HISTORY SQL command
- DeltaTable.history operation
spark.databricks.delta.commitInfo.enabled¶
CommitInfo
is added (logged) to a delta log only with spark.databricks.delta.commitInfo.enabled configuration property enabled.
Creating Empty CommitInfo¶
empty(
version: Option[Long] = None): CommitInfo
empty
...FIXME
empty
is used when:
DeltaHistoryManager
is requested to getCommitInfo
Creating CommitInfo¶
apply(
time: Long,
operation: String,
operationParameters: Map[String, String],
commandContext: Map[String, String],
readVersion: Option[Long],
isolationLevel: Option[String],
isBlindAppend: Option[Boolean],
operationMetrics: Option[Map[String, String]],
userMetadata: Option[String]): CommitInfo
apply
creates a CommitInfo
(for the given arguments and based on the given commandContext
for the user ID, user name, job, notebook, cluster).
commandContext
argument is always empty, but could be customized using ConvertToDeltaCommandBase.
apply
is used when:
OptimisticTransactionImpl
is requested to commit (with spark.databricks.delta.commitInfo.enabled configuration property enabled)DeltaCommand
is requested to commitLarge