Installation¶

Delta Lake is a Spark data source and as such installation boils down to using spark-submit's --packages command-line option with the following configuration properties for DeltaSparkSessionExtension and DeltaCatalog:

spark.sql.extensions (Spark SQL)
spark.sql.catalog.spark_catalog (Spark SQL)

Spark SQL Application¶

import org.apache.spark.sql.SparkSession
val spark = SparkSession
  .builder()
  .appName("...")
  .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
  .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
  .getOrCreate

Spark Shell¶

./bin/spark-shell \
  --packages io.delta:delta-core_2.12:0.8.0 \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

Last update: 2021-03-31