Skip to content

WriteIntoDelta

WriteIntoDelta Command

WriteIntoDelta is a <> that can write <> transactionally into a <>.

[[demo]] .Demo [source, scala]


import org.apache.spark.sql.delta.commands.WriteIntoDelta import org.apache.spark.sql.delta.DeltaLog import org.apache.spark.sql.SaveMode import org.apache.spark.sql.delta.DeltaOptions val tableName = "/tmp/delta/t1" val data = spark.range(5).toDF val writeCmd = WriteIntoDelta( deltaLog = DeltaLog.forTable(spark, tableName), mode = SaveMode.Overwrite, options = new DeltaOptions(Map.empty[String, String], spark.sessionState.conf), partitionColumns = Seq.empty[String], configuration = Map.empty[String, String], data)

// Review web UI @ http://localhost:4040

writeCmd.run(spark)

[[ImplicitMetadataOperation]] WriteIntoDelta is an <> of a <>.

[[RunnableCommand]] WriteIntoDelta is a logical command (RunnableCommand).

TIP: Read up on https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-LogicalPlan-RunnableCommand.html[RunnableCommand] in https://bit.ly/spark-sql-internals[The Internals of Spark SQL] online book.

WriteIntoDelta is <> when:

  • DeltaLog is requested to <> (when DeltaDataSource is requested to create a relation as a <> or a <>)

  • DeltaDataSource is requested to <> (as a <>)

== [[creating-instance]] Creating WriteIntoDelta Instance

WriteIntoDelta takes the following to be created:

  • [[deltaLog]] <>
  • [[mode]] SaveMode
  • [[options]] <>
  • [[partitionColumns]] Names of the partition columns (Seq[String])
  • [[configuration]] Configuration (Map[String, String])
  • [[data]] Data (DataFrame)

== [[run]] Running Command -- run Method

[source, scala]

run( sparkSession: SparkSession): Seq[Row]


NOTE: run is part of the RunnableCommand contract to run a command.

run requests the <> to <>.

run <> and requests the OptimisticTransaction to <>.

== [[write]] write Method

[source, scala]

write( txn: OptimisticTransaction, sparkSession: SparkSession): Seq[Action]


write checks out whether the write operation is to a delta table that already exists. If so (i.e. the <> of the transaction is above -1), write branches per the <>:

  • For ErrorIfExists, write throws an AnalysisException: +

    [path] already exists.
    

  • For Ignore, write does nothing

  • For Overwrite, write requests the <> to <>

write <>.

write...FIXME

NOTE: write is used exclusively when WriteIntoDelta is requested to <>.


Last update: 2020-09-24