DataStreamReader¶

DataStreamReader is an interface that Spark developers use to describe how to load data from a streaming data source.

Accessing DataStreamReader¶

DataStreamReader is available using SparkSession.readStream method.

import org.apache.spark.sql.SparkSession
val spark: SparkSession = ...

val streamReader = spark.readStream

Built-in Formats¶

DataStreamReader supports loading streaming data from the following built-in data sources:

• csv
• json
• orc
• parquet (default)
• text
• textFile

Loading Data¶

load(): DataFrame
load(
  path: String): DataFrame

DataStreamReader gives specialized methods for built-in data systems and formats.

In order to plug in a custom data source, DataStreamReader gives format and load methods.

load creates a streaming DataFrame that represents a "loading" streaming data node (and is internally a logical plan with a StreamingRelationV2 or StreamingRelation leaf logical operators).

load uses DataSource.lookupDataSource utility to look up the data source by source alias.

SupportsRead Tables with MICRO_BATCH_READ or CONTINUOUS_READ¶

For a TableProvider (that is not a FileDataSourceV2), load requests it for a Table.

For a Table with SupportsRead with MICRO_BATCH_READ or CONTINUOUS_READ capabilities, load creates a DataFrame with StreamingRelationV2 leaf logical operator.

If the DataSource is a StreamSourceProvider, load creates the StreamingRelationV2 with a StreamingRelation leaf logical operator.

For other Tables, load creates a DataFrame with a StreamingRelation leaf logical operator.

Other Data Sources¶

load creates a DataFrame with a StreamingRelation leaf logical operator.