Skip to content

DataWritingCommand -- Logical Commands That Write Query Data

DataWritingCommand is an <> of the <> for <> that write the result of executing <> (query data) to a relation when <>.

DataWritingCommand is resolved to a <> physical operator when BasicOperators execution planning strategy is executed (i.e. plan a <> to a <>).

[[contract]] .DataWritingCommand Contract [cols="1m,2",options="header",width="100%"] |=== | Property | Description

| outputColumnNames a| [[outputColumnNames]]

[source, scala]

outputColumnNames: Seq[String]

The output column names of the <>

Used when DataWritingCommand is requested for the <>

| query a| [[query]]

[source, scala]

query: LogicalPlan

The analyzed <> representing the data to write (i.e. whose result will be inserted into a relation)

Used when DataWritingCommand is requested for the <> and <>.

| run a| [[run]]

[source, scala]

run( sparkSession: SparkSession, child: SparkPlan): Seq[Row]

Executes the command to write query data (the result of executing[structured query])

Used when:

[[children]] When requested for the <>, DataWritingCommand simply returns the <>.

[[extensions]] .DataWritingCommands (Direct Implementations and Extensions Only) [cols="1,2",options="header",width="100%"] |=== | DataWritingCommand | Description

|[CreateDataSourceTableAsSelectCommand] | [[CreateDataSourceTableAsSelectCommand]]

| hive/[CreateHiveTableAsSelectCommand] | [[CreateHiveTableAsSelectCommand]]

| InsertIntoHadoopFsRelationCommand | [[InsertIntoHadoopFsRelationCommand]]

| hive/[SaveAsHiveFile] | [[SaveAsHiveFile]] Commands that write query result as Hive files (i.e. hive/[InsertIntoHiveDirCommand] and hive/[InsertIntoHiveTable])


=== [[basicWriteJobStatsTracker]] basicWriteJobStatsTracker Method

[source, scala]

basicWriteJobStatsTracker(hadoopConf: Configuration): BasicWriteJobStatsTracker

basicWriteJobStatsTracker simply creates and returns a new BasicWriteJobStatsTracker (with the given Hadoop Configuration and the <>).


basicWriteJobStatsTracker is used when:

  • SaveAsHiveFile is requested to <> (when hive/[InsertIntoHiveDirCommand] and hive/[InsertIntoHiveTable] logical commands are executed)

* InsertIntoHadoopFsRelationCommand logical command is executed

=== [[outputColumns]] Output Columns -- outputColumns Method

[source, scala]

outputColumns: Seq[Attribute]



outputColumns is used when:

  • hive/[CreateHiveTableAsSelectCommand], hive/[InsertIntoHiveDirCommand] and InsertIntoHadoopFsRelationCommand logical commands are executed

* SaveAsHiveFile is requested to <>

Performance Metrics

Key Name (in web UI) Description
numFiles number of written files
numOutputBytes bytes of written output
numOutputRows number of output rows
numParts number of dynamic part
Back to top