SupportsPushDownFilters¶

SupportsPushDownFilters is an extension of the ScanBuilder abstraction for scan builders that can pushFilters and pushedFilters (for filter pushdown performance optimization and thus reduce the size of the data to be read).

Contract¶

pushedFilters¶

Filter[] pushedFilters()

Data source filters that were pushed down to the data source (in pushFilters)

Used when V2ScanRelationPushDown logical optimization is executed (that uses PushDownUtils utility to pushFilters)

pushFilters¶

Filter[] pushFilters(
  Filter[] filters)

Data source filters that need to be evaluated again after scanning (so Spark can plan an extra filter operator)

Used when:

• V2ScanRelationPushDown logical optimization is executed (that uses PushDownUtils utility to pushFilters)
• CSVScanBuilder is requested for a Scan
• OrcScanBuilder is requested for a Scan

Implementations¶

• CSVScanBuilder
• OrcScanBuilder
• ParquetScanBuilder