Skip to content

Operation

Operation is an abstraction of operations that can be executed on a Delta table.

Operation is described by a name and parameters (that are simply used to create a CommitInfo for OptimisticTransactionImpl when committed and, as a way to bypass a transaction, ConvertToDeltaCommand).

Operation may have performance metrics.

Contract

Parameters

parameters: Map[String, Any]

Parameters of the operation (to create a CommitInfo with the JSON-encoded values)

Used when:

Implementations

Sealed Abstract Class

Operation is a Scala sealed abstract class which means that all of the implementations are in the same compilation unit (a single file).

AddColumns

Name: ADD COLUMNS

Parameters:

  • columns

Used when:

AddConstraint

ChangeColumn

Name Parameters changesData
CHANGE COLUMN
  • column
  • position (optional)
false

Used when:

Convert

CreateTable

Delete

DropConstraint

ManualUpdate

Merge

Name: MERGE

Parameters:

  • predicate
  • matchedPredicates
  • notMatchedPredicates

changesData: true

Used when:

Optimize

Name Parameters changesData
OPTIMIZE predicate true

Used when:

  • OptimizeExecutor is requested to optimize

RenameColumn

Name Parameters changesData
RENAME COLUMN
  • oldColumnPath
  • newColumnPath
false

Used when:

ReplaceColumns

ReplaceTable

SetTableProperties

Name: SET TBLPROPERTIES

Parameters:

  • properties

Used when:

StreamingUpdate

Name: STREAMING UPDATE

Parameters:

  • outputMode
  • queryId
  • epochId

Used when:

Truncate

Name Parameters changesData
TRUNCATE No parameters true

UnsetTableProperties

Update

UpdateColumnMetadata

UpdateSchema

UpgradeProtocol

Write

Creating Instance

Operation takes the following to be created:

  • Name of this operation
Abstract Class

Operation is an abstract class and cannot be created directly. It is created indirectly for the concrete Operations.

Serializing Parameter Values (to JSON Format)

jsonEncodedValues: Map[String, String]

jsonEncodedValues converts the values of the parameters to JSON format.

jsonEncodedValues is used when:

  • OptimisticTransactionImpl is requested to commit
  • ConvertToDeltaCommand command is requested to streamWrite

Operation Metrics

operationMetrics: Set[String]

operationMetrics is empty by default (and is expected to be overriden by concrete operations).

operationMetrics is used when:

Transforming Performance Metrics

transformMetrics(
  metrics: Map[String, SQLMetric]): Map[String, String]

transformMetrics returns a collection of SQLMetrics (Spark SQL) and their values (as text) that are defined as the operation metrics.

transformMetrics is used when:

User Metadata

userMetadata: Option[String]

userMetadata is undefined (None) by default (and is expected to be overriden by concrete operations).

userMetadata is used when:

changesData Flag

changesData: Boolean

changesData is disabled (false) by default (and is expected to be overriden by concrete operations).

Note

changesData seems not used.