Skip to content

AppendData Logical Command

AppendData is a V2WriteCommand that represents appending data (the result of executing a structured query) to a table (with the columns matching by name or position).

Note

AppendData has replaced the deprecated WriteToDataSourceV2 logical operator.

Creating Instance

AppendData takes the following to be created:

AppendData is created using byName and byPosition operators.

isByName flag

AppendData is given isByName flag when created:

isByName is part of the V2WriteCommand abstraction.

byName

byName(
  table: NamedRelation,
  df: LogicalPlan,
  writeOptions: Map[String, String] = Map.empty): AppendData

byName creates a AppendData with the isByName flag enabled (true).

byName is used when:

  • DataFrameWriter is requested to saveInternal (with SaveMode.Append mode) and saveAsTable (with SaveMode.Append mode)
  • DataFrameWriterV2 is requested to append

byPosition

byPosition(
  table: NamedRelation,
  query: LogicalPlan,
  writeOptions: Map[String, String] = Map.empty): AppendData

byPosition creates a AppendData with the isByName flag disabled (false).

byPosition is used when:

Execution Planning

AppendData is planned as one of the physical operators by DataSourceV2Strategy execution planning strategy:

  • AppendDataExecV1
  • AppendDataExec