OptimisticTransaction

OptimisticTransaction is a concrete OptimisticTransactionImpl (which seems more of a class name change than anything more important).

OptimisticTransaction is created when DeltaLog is used for the following:

When OptimisticTransaction (as a OptimisticTransactionImpl) is attempted to be committed (that does doCommit internally), the LogStore (of the DeltaLog) is requested to write actions to a delta file, e.g. _delta_log/00000000000000000001.json for the attempt version 1. Only when a FileAlreadyExistsException is thrown a commit is considered unsuccessful and retried.

import org.apache.spark.sql.delta.DeltaLog
val deltaLog = DeltaLog.forTable(spark, "/tmp/delta/users")
val txn = deltaLog.startTransaction

OptimisticTransaction can be associated with a thread as an active transaction.

Enable ALL logging level for org.apache.spark.sql.delta.OptimisticTransaction logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.delta.OptimisticTransaction=ALL

Refer to Logging.

Creating OptimisticTransaction Instance

OptimisticTransaction takes the following to be created:

The DeltaLog and Snapshot are part of the OptimisticTransactionImpl contract (which in turn inherits them from the TransactionalWrite contract and changes to val from def).

Active Thread-Local OptimisticTransaction — active Internal Registry

active: ThreadLocal[OptimisticTransaction]

active is a Java ThreadLocal with the OptimisticTransaction of the current thread.

ThreadLocal provides thread-local variables. These variables differ from their normal counterparts in that each thread that accesses one (via its get or set method) has its own, independently initialized copy of the variable.

ThreadLocal instances are typically private static fields in classes that wish to associate state with a thread (e.g., a user ID or Transaction ID).

active is assigned to the current thread using setActive utility and cleared in clearActive.

active is available using getActive utility.

There can only be one active OptimisticTransaction (or an IllegalStateException is thrown).

setActive Utility

setActive(
  txn: OptimisticTransaction): Unit

setActive simply associates the given OptimisticTransaction as active with the current thread.

setActive throws an IllegalStateException if there is an active OptimisticTransaction already associated:

Cannot set a new txn as active when one is already active
setActive is used exclusively when DeltaLog is requested to execute an operation in a new transaction.

Clearing Active Transaction — clearActive Utility

clearActive(): Unit

clearActive simply clears the active transaction (so no transaction is associated with a thread).

clearActive is used exclusively when DeltaLog is requested to execute an operation in a new transaction.

getActive Utility

getActive(): Option[OptimisticTransaction]

getActive simply returns the active transaction.

getActive seems unused.