StreamingRelationV2 Leaf Logical Operator¶

StreamingRelationV2 is a leaf logical operator that represents SupportsRead streaming tables (with MICRO_BATCH_READ or CONTINUOUS_READ capabilities) in a logical plan of a streaming query.

StreamingRelationV2 is a MultiInstanceRelation (Spark SQL).

StreamingRelationV2 is a ExposesMetadataColumns (Spark SQL).

Leaf Logical Operators

Learn more about Leaf Logical Operators, SupportsRead and Table Capabilities in The Internals of Spark SQL online book.

Creating Instance¶

StreamingRelationV2 takes the following to be created:

Source TableProvider (Spark SQL)
Source Name
Table (Spark SQL)
Extra Options
Output Attributes (Spark SQL)
CatalogPlugin (Spark SQL)
Identifier
V1 Relation LogicalPlan (Spark SQL)

StreamingRelationV2 is created when:

RelationResolution (Spark SQL) is requested to resolve a relation
StreamingRelationV2 is requested to withMetadataColumns
ResolveDataSource (Spark SQL) logical analysis rule is executed (for a SupportsRead table with MICRO_BATCH_READ or CONTINUOUS_READ capabilities)
MemoryStreamBase is requested for a logical query plan

Logical Resolution¶

StreamingRelationV2 is resolved to the following leaf logical operators:

StreamingDataSourceV2Relation or StreamingExecutionRelation when MicroBatchExecution stream execution engine is requested for an analyzed logical plan
StreamingDataSourceV2Relation when ContinuousExecution stream execution engine is created (and initializes an analyzed logical plan)

Metadata Columns¶

LogicalPlan

metadataOutput: Seq[AttributeReference]

metadataOutput is part of the LogicalPlan (Spark SQL) abstraction.

metadataOutput checks out whether this Table is SupportsMetadataColumns (Spark SQL) for the extra metadata columns.

Otherwise, metadataOutput returns no metadata columns (Nil).

Lazy Value

metadataOutput is a Scala lazy value to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards.

Learn more in the Scala Language Specification.

Add Metadata Columns to Output Columns¶

ExposesMetadataColumns

withMetadataColumns(): StreamingRelationV2

withMetadataColumns is part of the ExposesMetadataColumns (Spark SQL) abstraction.

withMetadataColumns determines whether thare are any extra metadata columns to be added to this output.

If so, withMetadataColumns creates a new StreamingRelationV2 with the extra metadata columns added. Otherwise, withMetadataColumns does nothing.

isStreaming¶

LogicalPlan

isStreaming: Boolean

isStreaming is part of the LogicalPlan (Spark SQL) abstraction.

isStreaming is always enabled (true).