FilePartitionReaderFactory¶

FilePartitionReaderFactory is an extension of the PartitionReaderFactory abstraction for PartitionReader factories of file-based connectors.

Contract¶

Building PartitionReader¶

buildReader(
  partitionedFile: PartitionedFile): PartitionReader[InternalRow]

PartitionReader (of InternalRows)

See:

• ParquetPartitionReaderFactory

Used when:

• FilePartitionReaderFactory is requested to create a reader

Building Columnar PartitionReader¶

buildColumnarReader(
  partitionedFile: PartitionedFile): PartitionReader[ColumnarBatch]

PartitionReader (of ColumnarBatchs)

See:

• ParquetPartitionReaderFactory

Used when:

• FilePartitionReaderFactory is requested to create a columnar reader

Options¶

options: FileSourceOptions

See:

• ParquetPartitionReaderFactory

Used when:

• FilePartitionReaderFactory is requested to create a reader and columnar reader

Implementations¶

• AvroPartitionReaderFactory
• CSVPartitionReaderFactory
• JsonPartitionReaderFactory
• OrcPartitionReaderFactory
• ParquetPartitionReaderFactory
• TextPartitionReaderFactory

Creating PartitionReader¶

createReader(
  partition: InputPartition): PartitionReader[InternalRow]

createReader...FIXME

Creating Columnar PartitionReader¶

createColumnarReader(
  partition: InputPartition): PartitionReader[ColumnarBatch]

createColumnarReader makes sure that the given InputPartition is a FilePartition (or throws an AssertionError).

createColumnarReader creates a new columnar PartitionReader for every PartitionedFile (of the FilePartition).

In the end, createColumnarReader creates a FilePartitionReader for the files.