Skip to content

JdbcRelationProvider

JdbcRelationProvider is used as a RelationProvider and a CreatableRelationProvider of the JDBC Data Source.

DataSourceRegister

JdbcRelationProvider is the DataSourceRegister to handle jdbc data source format.

Note

JdbcRelationProvider uses META-INF/services/org.apache.spark.sql.sources.DataSourceRegister file for registration that is available in the source code of Apache Spark.

Creating BaseRelation

createRelation(
  sqlContext: SQLContext,
  parameters: Map[String, String]): BaseRelation

createRelation creates a JDBCOptions (with the given parameters).

createRelation gets the schema (by querying the database system).

createRelation creates column partitions.

In the end, createRelation creates a JDBCRelation.

createRelation is part of the RelationProvider abstraction.

Writing Rows of Structured Query (DataFrame) to Table Using JDBC

createRelation(
  sqlContext: SQLContext,
  mode: SaveMode,
  parameters: Map[String, String],
  df: DataFrame): BaseRelation

createRelation is part of the CreatableRelationProvider abstraction.

Internally, createRelation creates a JDBCOptions (from the input parameters).

createRelation...FIXME

createRelation checks whether the table (given dbtable and url options in the input parameters) exists.

NOTE: createRelation uses a database-specific JdbcDialect to check whether a table exists.

createRelation branches off per whether the table already exists in the database or not.

If the table does not exist, createRelation creates the table (by executing CREATE TABLE with createTableColumnTypes and createTableOptions options from the input parameters) and writes the rows to the database in a single transaction.

If however the table does exist, createRelation branches off per SaveMode (see the following createRelation and SaveMode).

[[createRelation-CreatableRelationProvider-SaveMode]] .createRelation and SaveMode [cols="1,2",options="header",width="100%"] |=== | Name | Description

| Append | Saves the records to the table.

| ErrorIfExists a| Throws a AnalysisException with the message:

Table or view '[table]' already exists. SaveMode: ErrorIfExists.

| Ignore | Does nothing.

| Overwrite a| Truncates or drops the table

NOTE: createRelation truncates the table only when truncate JDBC option is enabled and JdbcDialect.md#isCascadingTruncateTable[isCascadingTruncateTable] is disabled. |===

In the end, createRelation closes the JDBC connection to the database and creates a JDBCRelation.