OptimizeTableCommand¶

OptimizeTableCommand is an OptimizeTableCommandBase for the following high-level operators:

• DeltaOptimizeBuilder.executeCompaction
• DeltaOptimizeBuilder.executeZOrderBy
• OPTIMIZE SQL command
• REORG TABLE SQL command

OptimizeTableCommand uses OptimizeExecutor to optimize (when executed).

OptimizeTableCommand is a transactional command (and starts a transaction at the beginning of execution).

OptimizeTableCommand is a UnaryNode (Spark SQL).

Creating Instance¶

OptimizeTableCommand takes the following to be created:

• Child LogicalPlan (Spark SQL)
• User-Defined Partition Predicates
• DeltaOptimizeContext
• zOrderBy attributes (aka interleaveBy attributes)

OptimizeTableCommand is created when:

• DeltaOptimizeBuilder is requested to execute
• DeltaReorgTableCommand is requested to optimizeByReorg
• OptimizeTableCommand is requested to apply

Creating OptimizeTableCommand¶

apply(
  path: Option[String],
  tableIdentifier: Option[TableIdentifier],
  userPartitionPredicates: Seq[String],
  optimizeContext: DeltaOptimizeContext = DeltaOptimizeContext())(
  zOrderBy: Seq[UnresolvedAttribute]): OptimizeTableCommand

apply creates an OptimizeTableCommand for a delta table (based on the given path or tableIdentifier).

apply is used when:

• DeltaSqlAstBuilder is requested to parse OPTIMIZE command

Executing Command¶

run(
  sparkSession: SparkSession): Seq[Row]

run extracts the delta table from the child logical plan.

run starts a transaction (for this DeltaTableV2).

run...FIXME

run validates the zOrderBy columns (that may throw DeltaIllegalArgumentException or DeltaAnalysisException exceptions and so break the command execution).

In the end, run creates an OptimizeExecutor to run optimization (with the given userPartitionPredicates, the zOrderBy attributes).