V2TableWriteExec Unary Physical Commands¶

V2TableWriteExec is an extension of the V2CommandExec abstraction for unary physical commands that writeWithV2.

Contract¶

Physical Query Plan¶

query: SparkPlan

SparkPlan for the data to be written out

Implementations¶

• TableWriteExecHelper
• V2ExistingTableWriteExec
• WriteToDataSourceV2Exec (Spark Structured Streaming)

writeWithV2¶

writeWithV2(
  batchWrite: BatchWrite): Seq[InternalRow]

writeWithV2 requests the physical query plan to execute (and produce a RDD[InternalRow]).

writeWithV2 requests the given BatchWrite to create a DataWriterFactory (with the number of partitions of the RDD)

writeWithV2 prints out the following INFO message to the logs:

Start processing data source write support: [batchWrite]. The input RDD has [n] partitions.

writeWithV2 runs a Spark job (Spark Core) with the DataWritingSparkTask for every partition. writeWithV2 requests the BatchWrite to onDataWriterCommit (with the result WriterCommitMessage) after every partition has been processed successfully.

writeWithV2 prints out the following INFO message to the logs:

Data source write support [batchWrite] is committing.

writeWithV2 requests the BatchWrite to commit (with all the result WriterCommitMessages).

writeWithV2 prints out the following INFO message to the logs:

Data source write support [batchWrite] committed.

In the end, writeWithV2 returns an empty collection (of InternalRows).

writeWithV2 is used when:

• TableWriteExecHelper is requested to writeToTable
• V2ExistingTableWriteExec is executed
• WriteToDataSourceV2Exec (Spark Structured Streaming) is executed

Logging¶

V2TableWriteExec is a Scala trait and logging is configured using the logger of the implementations.