StreamingRelationStrategy Execution Planning Strategy¶
StreamingRelationStrategy is an execution planning strategy that can plan streaming queries with StreamingRelation, StreamingExecutionRelation, and StreamingRelationV2 logical operators to StreamingRelationExec physical operators.
StreamingRelationStrategy is used when IncrementalExecution is requested to plan a streaming query.
Accessing StreamingRelationStrategy¶
StreamingRelationStrategy is available using SessionState (of a SparkSession).
spark.sessionState.planner.StreamingRelationStrategy
Demo¶
val rates = spark.
readStream.
format("rate").
load // <-- gives a streaming Dataset with a logical plan with StreamingRelation logical operator
// StreamingRelation logical operator for the rate streaming source
scala> println(rates.queryExecution.logical.numberedTreeString)
00 StreamingRelation DataSource(org.apache.spark.sql.SparkSession@31ba0af0,rate,List(),None,List(),None,Map(),None), rate, [timestamp#0, value#1L]
// StreamingRelationExec physical operator (shown without "Exec" suffix)
scala> rates.explain
== Physical Plan ==
StreamingRelation rate, [timestamp#0, value#1L]
// Let's do the planning manually
import spark.sessionState.planner.StreamingRelationStrategy
val physicalPlan = StreamingRelationStrategy.apply(rates.queryExecution.logical).head
scala> println(physicalPlan.numberedTreeString)
00 StreamingRelation rate, [timestamp#0, value#1L]