Skip to content


WriteIntoDelta Command

WriteIntoDelta is a <> that can write <> transactionally into a <>.

[[demo]] .Demo [source, scala]

import import import org.apache.spark.sql.SaveMode import val tableName = "/tmp/delta/t1" val data = spark.range(5).toDF val writeCmd = WriteIntoDelta( deltaLog = DeltaLog.forTable(spark, tableName), mode = SaveMode.Overwrite, options = new DeltaOptions(Map.empty[String, String], spark.sessionState.conf), partitionColumns = Seq.empty[String], configuration = Map.empty[String, String], data)

// Review web UI @ http://localhost:4040

[[ImplicitMetadataOperation]] WriteIntoDelta is an <> of a <>.

[[RunnableCommand]] WriteIntoDelta is a logical command (RunnableCommand).

TIP: Read up on[RunnableCommand] in[The Internals of Spark SQL] online book.

WriteIntoDelta is <> when:

  • DeltaLog is requested to <> (when DeltaDataSource is requested to create a relation as a <> or a <>)

  • DeltaDataSource is requested to <> (as a <>)

== [[creating-instance]] Creating WriteIntoDelta Instance

WriteIntoDelta takes the following to be created:

  • [[deltaLog]] <>
  • [[mode]] SaveMode
  • [[options]] <>
  • [[partitionColumns]] Names of the partition columns (Seq[String])
  • [[configuration]] Configuration (Map[String, String])
  • [[data]] Data (DataFrame)

== [[run]] Running Command -- run Method

[source, scala]

run( sparkSession: SparkSession): Seq[Row]

NOTE: run is part of the RunnableCommand contract to run a command.

run requests the <> to <>.

run <> and requests the OptimisticTransaction to <>.

== [[write]] write Method

[source, scala]

write( txn: OptimisticTransaction, sparkSession: SparkSession): Seq[Action]

write checks out whether the write operation is to a delta table that already exists. If so (i.e. the <> of the transaction is above -1), write branches per the <>:

  • For ErrorIfExists, write throws an AnalysisException: +

    [path] already exists.

  • For Ignore, write does nothing

  • For Overwrite, write requests the <> to <>

write <>.


NOTE: write is used exclusively when WriteIntoDelta is requested to <>.

Last update: 2020-09-24