DataFrameWriterV2¶

DataFrameWriterV2 is an API for Spark SQL developers to describe how to write a Dataset to an external storage using the DataSource V2.

DataFrameWriterV2 is a CreateTableWriter (and thus a WriteConfigMethods).

Demo¶

val nums = spark.range(5)
scala> :type nums
org.apache.spark.sql.Dataset[Long]

val writerV2 = nums.writeTo("t1")
scala> :type writerV2
org.apache.spark.sql.DataFrameWriterV2[Long]

Creating Instance¶

DataFrameWriterV2 takes the following to be created:

Name of the target table (multi-part table identifier)
Dataset

DataFrameWriterV2 is created when:

Dataset.writeTo operator is used

append¶

append(): Unit

append loads the table from the catalog and identifier (based on the table name).

When found, append creates an AppendData logical command (by name) with a DataSourceV2Relation logical operator (for the catalog and the identifier).

In the end, append runs the command with append execution name.

Tip

Execution is announced as a SparkListenerSQLExecutionEnd.

append throws a NoSuchTableException when the table could not be found.

scala> spark.range(5).writeTo("my.catalog.t1").append
org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table my.catalog.t1 not found;
  at org.apache.spark.sql.DataFrameWriterV2.append(DataFrameWriterV2.scala:162)
  ... 47 elided

create¶

create(): Unit

create is part of the CreateTableWriter abstraction.

create...FIXME

createOrReplace¶

createOrReplace(): Unit

createOrReplace is part of the CreateTableWriter abstraction.

createOrReplace...FIXME

option¶

option(
  key: String,
  value: String): DataFrameWriterV2[T]

option is part of the WriteConfigMethods abstraction.

option...FIXME

options¶

options(
  options: Map[String, String]): DataFrameWriterV2[T]

options is part of the WriteConfigMethods abstraction.

options...FIXME

overwrite¶

overwrite(
  condition: Column): Unit

overwrite...FIXME

overwritePartitions¶

overwritePartitions(): Unit

overwritePartitions...FIXME

partitionedBy¶

partitionedBy(
  column: Column,
  columns: Column*): CreateTableWriter[T]

partitionedBy is part of the CreateTableWriter abstraction.

partitionedBy...FIXME

replace¶

replace(): Unit

replace is part of the CreateTableWriter abstraction.

replace...FIXME

tableProperty¶

tableProperty(
  property: String,
  value: String): CreateTableWriter[T]

tableProperty is part of the CreateTableWriter abstraction.

tableProperty...FIXME

using¶

using(
  provider: String): CreateTableWriter[T]

using is part of the CreateTableWriter abstraction.

using...FIXME

Private Methods¶

internalReplace¶

internalReplace(
  orCreate: Boolean): Unit

internalReplace...FIXME

Executing Logical Command¶

runCommand(
  name: String)(
  command: LogicalPlan): Unit

runCommand...FIXME