StreamingDataSourceV2Relation Logical Operator¶

StreamingDataSourceV2Relation is a leaf logical operator that represents StreamingRelationV2 logical operator (with tables with a SupportsRead and MICRO_BATCH_READ or CONTINUOUS_READ capabilities) at execution time.

Tip

Learn more about Leaf Logical Operators, SupportsRead and Table Capabilities in The Internals of Spark SQL online book.

Creating Instance¶

StreamingDataSourceV2Relation takes the following to be created:

Output Attributes (Spark SQL)
Scan (Spark SQL)
SparkDataStream
Start Offset (default: undefined)
End Offset (default: undefined)

StreamingDataSourceV2Relation is created when:

MicroBatchExecution stream execution engine is requested for an analyzed logical query plan (for StreamingRelationV2 with a SupportsRead table with MICRO_BATCH_READ capability)
ContinuousExecution stream execution engine is requested for an analyzed logical query plan (for StreamingRelationV2 with a SupportsRead table with CONTINUOUS_READ capability)

Computing Stats¶

computeStats(): Statistics

For Scans with SupportsReportStatistics, computeStats requests the scan to estimateStatistics.

Tip

Learn more about Scan and SupportsReportStatistics in The Internals of Spark SQL online book.

For other types of scans, computeStats simply assumes the default size and no row count.

computeStats is part of the LeafNode abstraction.