TransactionalWrite¶

TransactionalWrite is an abstraction of optimistic transactional writers that can write a structured query out to a Delta table.

Contract¶

DeltaLog¶

deltaLog: DeltaLog

DeltaLog (of a delta table) that this transaction is changing

Used when:

• ActiveOptimisticTransactionRule logical rule is executed
• OptimisticTransactionImpl is requested to prepare a commit, doCommit, checkAndRetry, and perform post-commit operations (and execute delta log checkpoint)
• ConvertToDeltaCommand is executed
• DeltaCommand is requested to buildBaseRelation and commitLarge
• MergeIntoCommand is executed
• TransactionalWrite is requested to write a structured query out to a delta table
• GenerateSymlinkManifest post-commit hook is executed
• ImplicitMetadataOperation is requested to updateMetadata
• DeltaSink is requested to addBatch

Metadata¶

metadata: Metadata

Metadata (of the delta table) that this transaction is changing

Protocol¶

protocol: Protocol

Protocol (of the delta table) that this transaction is changing

Used when:

• OptimisticTransactionImpl is requested to updateMetadata, verifyNewMetadata and prepareCommit
• ConvertToDeltaCommand is executed

Snapshot¶

snapshot: Snapshot

Snapshot (of the delta table) that this transaction is reading at

Implementations¶

• OptimisticTransaction

hasWritten Flag¶

hasWritten: Boolean = false

TransactionalWrite uses hasWritten internal registry to prevent OptimisticTransactionImpl from updating metadata after having written out files.

hasWritten is initially false and changes to true after having written out files.

Writing Data Out (Result Of Structured Query)¶

writeFiles(
  data: Dataset[_]): Seq[AddFile]
writeFiles(
  data: Dataset[_],
  writeOptions: Option[DeltaOptions]): Seq[AddFile]
writeFiles(
  data: Dataset[_],
  isOptimize: Boolean): Seq[AddFile]
writeFiles(
  data: Dataset[_],
  writeOptions: Option[DeltaOptions],
  isOptimize: Boolean): Seq[AddFile]

writeFiles creates a DeltaInvariantCheckerExec and a DelayedCommitProtocol to write out files to the data path (of the DeltaLog).

writeFiles is executed within SQLExecution.withNewExecutionId.

In the end, writeFiles returns the addedStatuses of the DelayedCommitProtocol committer.

Internally, writeFiles turns the hasWritten flag on (true).

writeFiles normalize the given data dataset (based on the partitionColumns of the Metadata).

writeFiles getPartitioningColumns based on the partitionSchema of the Metadata.

DelayedCommitProtocol Committer¶

writeFiles creates a DelayedCommitProtocol committer for the data path of the DeltaLog.

writeFiles gets the invariants from the schema of the Metadata.

DeltaInvariantCheckerExec¶

writeFiles requests a new Execution ID (that is used to track all Spark jobs of FileFormatWriter.write in Spark SQL) with a physical query plan of a new DeltaInvariantCheckerExec unary physical operator (with the executed plan of the normalized query execution as the child operator).

Creating Committer¶

getCommitter(
  outputPath: Path): DelayedCommitProtocol

getCommitter creates a new DelayedCommitProtocol with the delta job ID and the given outputPath (and no random prefix).

getPartitioningColumns¶

getPartitioningColumns(
  partitionSchema: StructType,
  output: Seq[Attribute],
  colsDropped: Boolean): Seq[Attribute]

getPartitioningColumns...FIXME

normalizeData¶

normalizeData(
  data: Dataset[_],
  partitionCols: Seq[String]): (QueryExecution, Seq[Attribute])

normalizeData...FIXME

makeOutputNullable¶

makeOutputNullable(
  output: Seq[Attribute]): Seq[Attribute]

makeOutputNullable...FIXME

Usage¶

writeFiles is used when:

• DeleteCommand, MergeIntoCommand, UpdateCommand, and WriteIntoDelta commands are executed
• DeltaSink is requested to add a streaming micro-batch