Demo: replaceWhere¶

This demo shows replaceWhere predicate option.

In combination with Overwrite mode, a replaceWhere option can be used to transactionally replace data that matches a predicate.

Create Delta Table¶

val table = "d1"

sql(s"""
  CREATE TABLE $table (`id` LONG, p STRING)
  USING delta
  PARTITIONED BY (p)
  COMMENT 'Delta table'
""")

spark.catalog.listTables.show

+----+--------+-----------+---------+-----------+
|name|database|description|tableType|isTemporary|
+----+--------+-----------+---------+-----------+
|  d1| default|Delta table|  MANAGED|      false|
+----+--------+-----------+---------+-----------+

import org.apache.spark.sql.delta.DeltaLog
val dt = DeltaLog.forTable(spark, table)
val m = dt.snapshot.metadata
val partitionSchema = m.partitionSchema

Write Data¶

Seq((0L, "a"),
    (1L, "a"))
  .toDF("id", "p")
  .write
  .format("delta")
  .option("replaceWhere", "p = 'a'")
  .mode("overwrite")
  .saveAsTable(table)