Skip to content

DataSkippingReaderBase

DataSkippingReaderBase is an extension of the DeltaScanGenerator abstraction for DeltaScan generators.

Contract

Dataset of AddFiles

allFiles: Dataset[AddFile]

Used when:

DeltaLog

deltaLog: DeltaLog

Used when:

Metadata

metadata: Metadata

Used when:

numOfFiles

numOfFiles: Long

Used when:

Path

path: Path

Redacted Path

redactedPath: String

Used when:

Schema

schema: StructType

Used when:

sizeInBytes

sizeInBytes: Long

Used when:

version

version: Long

Used when:

Implementations

spark.databricks.delta.stats.skipping

useStats: Boolean

useStats is the value of spark.databricks.delta.stats.skipping configuration property.

useStats is used when:

filesForScan

filesForScan(
  projection: Seq[Attribute],
  filters: Seq[Expression]): DeltaScan // (1)!
filesForScan(
  projection: Seq[Attribute],
  filters: Seq[Expression],
  keepNumRecords: Boolean): DeltaScan
  1. keepNumRecords flag is false

filesForScan...FIXME

filesForScan is part of the DeltaScanGeneratorBase abstraction.

Back to top