Skip to content

Operation

Operation is an abstraction of operations that can be executed on Delta tables.

Operation is described by a name and parameters (that are simply used to create a CommitInfo for OptimisticTransactionImpl when committed and, as a way to bypass a transaction, ConvertToDeltaCommand).

Operation may have performance metrics.

Contract

Parameters

parameters: Map[String, Any]

Parameters of the operation (to create a CommitInfo with the JSON-encoded values)

Used when Operation is requested for parameters with the values in JSON format

Implementations

Sealed Abstract Class

Operation is a Scala sealed abstract class which means that all of the implementations are in the same compilation unit (a single file).

AddColumns

AddConstraint

ChangeColumn

Convert

CreateTable

Delete

DropConstraint

ManualUpdate

Merge

ReplaceColumns

ReplaceTable

SetTableProperties

Name: SET TBLPROPERTIES

Parameters:

  • properties

Used when:

StreamingUpdate

Name: STREAMING UPDATE

Parameters:

  • outputMode
  • queryId
  • epochId

Used when:

Truncate

UnsetTableProperties

Update

UpdateColumnMetadata

UpdateSchema

UpgradeProtocol

Write

Merge

Recorded when a merge operation is committed to a Delta table (when MergeIntoCommand is executed)

Creating Instance

Operation takes the following to be created:

  • Name of this operation
Abstract Class

Operation is an abstract class and cannot be created directly. It is created indirectly for the concrete Operations.

Serializing Parameter Values (to JSON Format)

jsonEncodedValues: Map[String, String]

jsonEncodedValues converts the values of the parameters to JSON format.

jsonEncodedValues is used when:

  • OptimisticTransactionImpl is requested to commit
  • ConvertToDeltaCommand command is requested to streamWrite

operationMetrics Registry

operationMetrics: Set[String]

operationMetrics is empty by default (and is expected to be overriden by concrete operations)

operationMetrics is used when Operation is requested to transformMetrics.

Transforming Performance Metrics

transformMetrics(
  metrics: Map[String, SQLMetric]): Map[String, String]

transformMetrics returns a collection of SQLMetrics (Spark SQL) and their values (as text) that are defined as the operationMetrics.

transformMetrics is used when:


Last update: 2021-06-16