Offset Logical Operator¶
Offset is an OrderPreservingUnaryNode that can skip the specified number of rows (offset) from the beginning of the output of the child logical operator.
Spark Structured Streaming
Offset is not supported on streaming DataFrames/Datasets.
Creating Instance¶
Offset takes the following to be created:
- Offset Expression
- Child Logical operator
Offset is created when:
AstBuilderis requested to withQueryResultClauses- Dataset.offset operator is used
Catalyst DSL¶
Catalyst DSL comes with offset operator to create an Offset.
offset(
offsetExpr: Expression): LogicalPlan
Logical Optimizations¶
Offset is optimized using the following logical optimizations:
EliminateOffsets- LimitPushDown
- V2ScanRelationPushDown
Physical Planning¶
Offset is planned as either GlobalLimitExec or CollectLimitExec physical operators.
maxRows¶
maxRows...FIXME