Skip to content

Offset Logical Operator

Offset is an OrderPreservingUnaryNode that can skip the specified number of rows (offset) from the beginning of the output of the child logical operator.

Spark Structured Streaming Unsupported

Offset is not supported on streaming DataFrames/Datasets (that is enforced by UnsupportedOperationChecker at QueryExecution).

Creating Instance

Offset takes the following to be created:

Offset is created when:

Catalyst DSL

Catalyst DSL comes with offset operator to create an Offset.

offset(
  offsetExpr: Expression): LogicalPlan

Logical Optimizations

Offset is optimized using the following logical optimizations:

Physical Planning

Offset is planned as either GlobalLimitExec or CollectLimitExec physical operators.

maxRows

LogicalPlan
maxRows: Option[Long]

maxRows is part of the LogicalPlan abstraction.

maxRows...FIXME