CachedBatchSerializer¶

CachedBatchSerializer is an abstraction of serializers of CachedBatches.

CachedBatchSerializer is configured using spark.sql.cache.serializer configuration property (for InMemoryRelation).

CachedBatchSerializer is used to create a CachedRDDBuilder.

CachedBatchSerializer is a Serializable.

Contract (Subset)¶

buildFilter¶

buildFilter(
  predicates: Seq[Expression],
  cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch]

See:

• SimpleMetricsCachedBatchSerializer

Used when:

• InMemoryTableScanExec physical operator is requested to filteredCachedBatches (with spark.sql.inMemoryColumnarStorage.partitionPruning enabled)

supportsColumnarInput¶

supportsColumnarInput(
  schema: Seq[Attribute]): Boolean

See:

• DefaultCachedBatchSerializer

Used when:

• CachedRDDBuilder is requested to buildBuffers
• InMemoryRelation is created

Implementations¶

• SimpleMetricsCachedBatchSerializer