WriteIntoDelta
WriteIntoDelta Command¶
WriteIntoDelta
is a <
[[demo]] .Demo [source, scala]
import org.apache.spark.sql.delta.commands.WriteIntoDelta import org.apache.spark.sql.delta.DeltaLog import org.apache.spark.sql.SaveMode import org.apache.spark.sql.delta.DeltaOptions val tableName = "/tmp/delta/t1" val data = spark.range(5).toDF val writeCmd = WriteIntoDelta( deltaLog = DeltaLog.forTable(spark, tableName), mode = SaveMode.Overwrite, options = new DeltaOptions(Map.empty[String, String], spark.sessionState.conf), partitionColumns = Seq.empty[String], configuration = Map.empty[String, String], data)
// Review web UI @ http://localhost:4040
writeCmd.run(spark)¶
[[ImplicitMetadataOperation]] WriteIntoDelta
is an <
[[RunnableCommand]] WriteIntoDelta
is a logical command (RunnableCommand
).
TIP: Read up on https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-LogicalPlan-RunnableCommand.html[RunnableCommand] in https://bit.ly/spark-sql-internals[The Internals of Spark SQL] online book.
WriteIntoDelta
is <
-
DeltaLog
is requested to <> (when DeltaDataSource
is requested to create a relation as a <> or a < >) -
DeltaDataSource
is requested to <> (as a < >)
== [[creating-instance]] Creating WriteIntoDelta Instance
WriteIntoDelta
takes the following to be created:
- [[deltaLog]] <
> - [[mode]]
SaveMode
- [[options]] <
> - [[partitionColumns]] Names of the partition columns (
Seq[String]
) - [[configuration]] Configuration (
Map[String, String]
) - [[data]] Data (
DataFrame
)
== [[run]] Running Command -- run
Method
[source, scala]¶
run( sparkSession: SparkSession): Seq[Row]
NOTE: run
is part of the RunnableCommand
contract to run a command.
run
requests the <
run
<OptimisticTransaction
to <
== [[write]] write
Method
[source, scala]¶
write( txn: OptimisticTransaction, sparkSession: SparkSession): Seq[Action]
write
checks out whether the write operation is to a delta table that already exists. If so (i.e. the <-1
), write
branches per the <
-
For
ErrorIfExists
,write
throws anAnalysisException
: +[path] already exists.
-
For
Ignore
,write
does nothing -
For
Overwrite
,write
requests the <> to < >
write
<
write
...FIXME
NOTE: write
is used exclusively when WriteIntoDelta
is requested to <