WriteToDataSourceV2 Logical Operator¶
WriteToDataSourceV2 is a unary logical operator (Spark SQL) that represents WriteToMicroBatchDataSource unary logical operator at logical optimization time for writing data in Spark Structured Streaming.
Deprecated
WriteToDataSourceV2 is deprecated since Spark SQL 2.4.0 (in favour of AppendData logical operator and alike).
Creating Instance¶
WriteToDataSourceV2 takes the following to be created:
-
DataSourceV2Relation(Spark SQL) -
BatchWrite(Spark SQL) - Logical Query Plan (Spark SQL)
- Write
CustomMetrics (Spark SQL)
WriteToDataSourceV2 is created when:
V2Writes(Spark SQL) logical optimization is requested to optimize a logical query (with a WriteToMicroBatchDataSource unary logical operator)
Query Planning¶
WriteToDataSourceV2 is planned as WriteToDataSourceV2Exec physical operator by DataSourceV2Strategy (Spark SQL) execution planning strategy.