DataWritingCommandExec Physical Operator¶
DataWritingCommandExec
is a UnaryExecNode that is the execution environment for a DataWritingCommand logical command at execution time.
Creating Instance¶
DataWritingCommandExec
takes the following to be created:
- DataWritingCommand
- Child SparkPlan
DataWritingCommandExec
is created when:
- BasicOperators execution planning strategy is requested to plan a DataWritingCommand logical command
Performance Metrics¶
metrics: Map[String, SQLMetric]
metrics
requests the DataWritingCommand for the metrics.
metrics
is part of the SparkPlan abstraction.
sideEffectResult¶
sideEffectResult: Seq[InternalRow]
Lazy Value
sideEffectResult
is a Scala lazy value to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards.
Learn more in the Scala Language Specification.
sideEffectResult
requests the DataWritingCommand to run (with the active SparkSession and the child logical operator) that produces output Rows.
In the end, sideEffectResult
creates a Catalyst converter (for the schema) to convert the output rows.
Used when:
DataWritingCommandExec
is requested to executeCollect, executeToIterator, executeTake, executeTail and doExecute
Executing Physical Operator¶
doExecute(): RDD[InternalRow]
doExecute
requests the SparkPlan to parallelize
the sideEffectResult (with 1
partition).
doExecute
is part of the SparkPlan abstraction.