DeltaGenerateCommand¶

DeltaGenerateCommand is a RunnableCommand (Spark SQL) to execute a generate function on a delta table.

DeltaGenerateCommand is used for the following:

• GENERATE SQL command
• DeltaTable.generate operation

DeltaGenerateCommand supports symlink_format_manifest mode name only.

Demo¶

val path = "/tmp/delta/d01"
val tid = s"delta.`$path`"
val q = s"GENERATE symlink_format_manifest FOR TABLE $tid"
sql(q).collect

Creating Instance¶

DeltaGenerateCommand takes the following to be created:

• Mode Name
• TableIdentifier (Spark SQL)

DeltaGenerateCommand is created for:

• GENERATE SQL command (that uses DeltaSqlAstBuilder to parse GENERATE SQL command)
• DeltaTable.generate operator (that uses DeltaTableOperations to executeGenerate)

Generate Mode Name¶

DeltaGenerateCommand is given a mode name when created.

DeltaGenerateCommand uses a lookup table of the supported generation functions by mode name (yet supports just symlink_format_manifest).

Executing Command¶

run(
  sparkSession: SparkSession): Seq[Row]

run is part of the RunnableCommand (Spark SQL) abstraction.

run creates a Hadoop Path to (the location of) the delta table (based on DeltaTableIdentifier).

run creates a DeltaLog for the delta table.

run executes the generation function for the mode name.

run returns no rows (an empty collection).

IllegalArgumentException¶

run throws an IllegalArgumentException when executed with an unsupported mode name:

Specified mode '[modeName]' is not supported. Supported modes are: [supportedModes]

AnalysisException¶

run throws an AnalysisException when executed for a non-delta table:

GENERATE is only supported for Delta tables.