StreamSourceProvider¶

StreamSourceProvider is an abstraction of data source providers that can create a streaming source for a data format or system.

StreamSourceProvider is part of Data Source API V1 for Micro-Batch Stream Processing.

Contract¶

Creating Streaming Source¶

createSource(
  sqlContext: SQLContext,
  metadataPath: String,
  schema: Option[StructType],
  providerName: String,
  parameters: Map[String, String]): Source

Creates a streaming Source

metadataPath is the value of the optional user-specified checkpointLocation option or resolved by StreamingQueryManager.

Used when:

• DataSource is requested to create a streaming source (for a StreamSourceProvider)

Source Schema¶

sourceSchema(
  sqlContext: SQLContext,
  schema: Option[StructType],
  providerName: String,
  parameters: Map[String, String]): (String, StructType)

Name and schema of the Streaming Source

Used when:

• DataSource is requested for metadata of a streaming source (when MicroBatchExecution is requested to initialize the analyzed logical plan)

Implementations¶

• KafkaSourceProvider