GroupBasedRowLevelOperationScanPlanning Logical Optimization¶

GroupBasedRowLevelOperationScanPlanning is a non-excludable logical optimization (Rule[LogicalPlan]) that is part of earlyScanPushDownRules of SparkOptimizer.

Executing Rule¶

Signature

apply(
  plan: LogicalPlan): LogicalPlan

apply is part of the Rule abstraction.

apply transforms ReplaceData with DataSourceV2Relations in the given LogicalPlan.

apply...FIXME

apply prints out the following INFO message to the logs:

Pushing operators to [relationName]
Pushed filters: [pushedFilters]
Filters that were not pushed: [remainingFilters]
Output: [output]

Logging¶

Enable ALL logging level for org.apache.spark.sql.execution.datasources.v2.GroupBasedRowLevelOperationScanPlanning logger to see what happens inside.

Add the following line to conf/log4j2.properties:

logger.GroupBasedRowLevelOperationScanPlanning.name = org.apache.spark.sql.execution.datasources.v2.GroupBasedRowLevelOperationScanPlanning
logger.GroupBasedRowLevelOperationScanPlanning.level = all

Refer to Logging.