Delta Connector (Spark Connector for Delta Lake)¶

Delta Connector (Spark Connector for Delta Lake) is an extension of Spark SQL (based on Connector API) to support batch and streaming queries over delta tables.

val rawDeltaTable = spark
  .read
  .format("delta")
  .load("raw_delta_table")

myDeltaTable
  .write
  .format("delta")
  .save("bronze_delta_table")

Options¶

Delta Connector defines options for reading and writing delta tables.

The options can be defined using option method of the following:

• For batch queries, DataFrameReader (Spark SQL) and DataFrameWriter (Spark SQL)
• For streaming queries, DataStreamReader (Spark Structured Streaming) and DataStreamWriter (Spark Structured Streaming)
• SQL queries

The options are available at runtime as DeltaOptions.