DeltaSparkSessionExtension

DeltaSparkSessionExtension is an extension of Spark SQL to activate DeltaSqlParser for Delta SQL commands.

DeltaSparkSessionExtension is registered using spark.sql.extensions configuration property while creating a SparkSession in a Spark application.

Registering DeltaSparkSessionExtension with Spark SQL Application
import org.apache.spark.sql.SparkSession
val spark = SparkSession
  .builder()
  .appName("...")
  .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
  .getOrCreate
Registering DeltaSparkSessionExtension with Spark Shell
/*
./bin/spark-shell \
  --packages io.delta:delta-core_2.12:0.5.0 \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
*/

assert(spark.conf.get("spark.sql.extensions") == "io.delta.sql.DeltaSparkSessionExtension")

// Create a delta table
val data = Seq((0,"zero")).toDF("id", "name")
// note the delta format
data.write.format("delta").partitionBy("id").save("/tmp/delta/t1")
import io.delta.tables.DeltaTable
assert(DeltaTable.isDeltaTable("/tmp/delta/t1"))

val describe = sql("DESCRIBE HISTORY delta.`/tmp/delta/t1`")
scala> describe.show
+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+
|version|          timestamp|userId|userName|operation| operationParameters| job|notebook|clusterId|readVersion|isolationLevel|isBlindAppend|
+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+
|      0|2019-12-03 11:40:27|  null|    null|    WRITE|[mode -> ErrorIfE...|null|    null|     null|       null|          null|         true|
+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+