DataWritingCommandExec Physical Operator¶
DataWritingCommandExec is a UnaryExecNode that is the execution environment for a DataWritingCommand logical command at execution time.
Creating Instance¶
DataWritingCommandExec takes the following to be created:
- DataWritingCommand
- Child SparkPlan
DataWritingCommandExec is created when:
- BasicOperators execution planning strategy is requested to plan a DataWritingCommand logical command
Performance Metrics¶
metrics: Map[String, SQLMetric]
metrics requests the DataWritingCommand for the metrics.
metrics is part of the SparkPlan abstraction.
sideEffectResult¶
sideEffectResult: Seq[InternalRow]
Lazy Value
sideEffectResult is a Scala lazy value to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards.
Learn more in the Scala Language Specification.
sideEffectResult requests the DataWritingCommand to run (with the active SparkSession and the child logical operator) that produces output Rows.
In the end, sideEffectResult creates a Catalyst converter (for the schema) to convert the output rows.
Used when:
DataWritingCommandExecis requested to executeCollect, executeToIterator, executeTake, executeTail and doExecute
Executing Physical Operator¶
doExecute(): RDD[InternalRow]
doExecute requests the SparkPlan to parallelize the sideEffectResult (with 1 partition).
doExecute is part of the SparkPlan abstraction.