DeltaTableOperations — Delta DML Operations

DeltaTableOperations is the abstraction of management services of a DeltaTable for executing delete, history, update, and vacuum commands.

DeltaTable is the default and only known DeltaTableOperations in Delta Lake.
Delete and Update operations do not support subqueries (and throw an AnalysisException otherwise).

Executing Generate Command — executeGenerate Method

  tblIdentifier: String,
  mode: String): Unit

executeGenerate requests the SQL parser (of the SparkSession) to parse the given table identifier, creates a DeltaGenerateCommand and runs it.

executeGenerate is used in DeltaTable.generate operator.

Executing Delete Command — executeDelete Method

  condition: Option[Expression]): Unit

executeDelete creates a QueryExecution for the Delete logical operator with (the analyzed logical plan of) the DeltaTable. executeDelete requests the QueryExecution for the analyzed logical plan that is (again?!) a Delete unary node.

FIXME What’s the purpose of all this resolutions?

In the end, executeDelete simply creates a DeleteCommand (for the resolved delete) and executes it.

executeDelete throws an AnalysisException when the condition expression contains subqueries.

executeDelete is used in DeltaTable.delete operator.

executeHistory Method

  deltaLog: DeltaLog,
  limit: Option[Int]): DataFrame


executeHistory is used when…​FIXME

executeUpdate Method

  set: Map[String, Column],
  condition: Option[Column]): Unit


executeUpdate is used when…​FIXME

Executing Vacuum Command — executeVacuum Method

  deltaLog: DeltaLog,
  retentionHours: Option[Double]): DataFrame

executeVacuum simply uses the VacuumCommand utility to gc (with the dryRun flag off) and returns an empty DataFrame.

executeVacuum returns an empty DataFrame not the one from VacuumCommand.gc.
executeVacuum is used exclusively in DeltaTable.vacuum operator.

makeUpdateTable Method

  target: DeltaTable,
  onCondition: Option[Column],
  setColumns: Seq[(String, Column)]): UpdateTable


makeUpdateTable is used when…​FIXME

subqueryNotSupportedCheck Internal Method

  condition: Option[Expression],
  op: String): Unit

subqueryNotSupportedCheck traverses the condition expression and throws an AnalysisException when it finds a SubqueryExpression:

Subqueries are not supported in the [op] (condition = [sql]).
subqueryNotSupportedCheck is used when DeltaTableOperations is requested to execute delete and update operations.