WriteIntoDelta

WriteIntoDelta Command¶

WriteIntoDelta is a <> that can write <> transactionally into a <>.

[[demo]] .Demo [source, scala]

import org.apache.spark.sql.delta.commands.WriteIntoDelta import org.apache.spark.sql.delta.DeltaLog import org.apache.spark.sql.SaveMode import org.apache.spark.sql.delta.DeltaOptions val tableName = "/tmp/delta/t1" val data = spark.range(5).toDF val writeCmd = WriteIntoDelta( deltaLog = DeltaLog.forTable(spark, tableName), mode = SaveMode.Overwrite, options = new DeltaOptions(Map.empty[String, String], spark.sessionState.conf), partitionColumns = Seq.empty[String], configuration = Map.empty[String, String], data)

// Review web UI @ http://localhost:4040

writeCmd.run(spark)¶

[[ImplicitMetadataOperation]] WriteIntoDelta is an <> of a <>.

[[RunnableCommand]] WriteIntoDelta is a logical command (RunnableCommand).

TIP: Read up on https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-LogicalPlan-RunnableCommand.html[RunnableCommand] in https://bit.ly/spark-sql-internals[The Internals of Spark SQL] online book.

WriteIntoDelta is <> when:

• DeltaLog is requested to <> (when DeltaDataSource is requested to create a relation as a <> or a <>)

• DeltaDataSource is requested to <> (as a <>)

== [[creating-instance]] Creating WriteIntoDelta Instance

WriteIntoDelta takes the following to be created:

• [[deltaLog]] <>
• [[mode]] SaveMode
• [[options]] <>
• [[partitionColumns]] Names of the partition columns (Seq[String])
• [[configuration]] Configuration (Map[String, String])
• [[data]] Data (DataFrame)

== [[run]] Running Command -- run Method

[source, scala]¶

run( sparkSession: SparkSession): Seq[Row]

NOTE: run is part of the RunnableCommand contract to run a command.

run requests the <> to <>.

run <> and requests the OptimisticTransaction to <>.

== [[write]] write Method

[source, scala]¶

write( txn: OptimisticTransaction, sparkSession: SparkSession): Seq[Action]

write checks out whether the write operation is to a delta table that already exists. If so (i.e. the <> of the transaction is above -1), write branches per the <>:

• For ErrorIfExists, write throws an AnalysisException: +

  [path] already exists.
  

• For Ignore, write does nothing

• For Overwrite, write requests the <> to <>

write <>.

write...FIXME

NOTE: write is used exclusively when WriteIntoDelta is requested to <>.