StreamingRelation Leaf Logical Operator¶

StreamingRelation is a leaf logical operator (Spark SQL) that represents a streaming source in a logical plan.

StreamingRelation is resolved (planned) to a StreamingExecutionRelation (right after StreamExecution starts running batches).

Creating Instance¶

StreamingRelation takes the following to be created:

DataSource
Short Name of the Streaming Source
Output Attributes (Seq[Attribute])

StreamingRelation is created when:

DataStreamReader is requested to load data from a streaming source and creates a streaming query.

StreamingRelation Represents Streaming Source

Creating StreamingRelation for DataSource¶

apply(
  dataSource: DataSource): StreamingRelation

apply creates a StreamingRelation for the given DataSource.

apply is used when:

DataStreamReader is requested for a streaming query

isStreaming¶

isStreaming: Boolean

isStreaming is part of the LogicalPlan (Spark SQL) abstraction.

isStreaming flag is always true.

import org.apache.spark.sql.execution.streaming.StreamingRelation
val relation = rate.queryExecution.logical.asInstanceOf[StreamingRelation]
assert(relation.isStreaming)

Text Respresentation¶

toString: String

toString gives the source name.

Demo¶

val rate = spark.
  readStream.
  format("rate").
  load("hello")

scala> println(rate.queryExecution.logical.numberedTreeString)
00 StreamingRelationV2 org.apache.spark.sql.execution.streaming.sources.RateStreamProvider@69ab1abc, rate, Map(path -> hello), [timestamp#0, value#1L]