V2TableWriteExec Unary Physical Commands¶
V2TableWriteExec
is an extension of the V2CommandExec abstraction for unary physical commands that writeWithV2.
Contract¶
Physical Query Plan¶
query: SparkPlan
SparkPlan for the data to be written out
Implementations¶
writeWithV2¶
writeWithV2(
batchWrite: BatchWrite): Seq[InternalRow]
writeWithV2
requests the physical query plan to execute (that gives a RDD[InternalRow]
).
writeWithV2
requests the given BatchWrite
for a DataWriterFactory.
writeWithV2
prints out the following INFO message to the logs:
Start processing data source write support: [batchWrite]. The input RDD has [n] partitions.
writeWithV2
runs a Spark job (Spark Core) with the DataWritingSparkTask for every partition. writeWithV2
requests the BatchWrite
to onDataWriterCommit (with the result WriterCommitMessage
) after every partition has been processed successfully.
writeWithV2
prints out the following INFO message to the logs:
Data source write support [batchWrite] is committing.
writeWithV2
requests the BatchWrite
to commit (with all the result WriterCommitMessage
s).
writeWithV2
prints out the following INFO message to the logs:
Data source write support [batchWrite] committed.
In the end, writeWithV2
returns no InternalRow
s.
writeWithV2
is used when:
WriteToDataSourceV2Exec
is executedV2ExistingTableWriteExec
is executedTableWriteExecHelper
is requested to writeToTable