FilePartitionReaderFactory¶
FilePartitionReaderFactory is an extension of the PartitionReaderFactory abstraction for PartitionReader factories of file-based connectors.
Contract¶
Building PartitionReader¶
buildReader(
partitionedFile: PartitionedFile): PartitionReader[InternalRow]
PartitionReader (of InternalRows)
See:
Used when:
FilePartitionReaderFactoryis requested to create a reader
Building Columnar PartitionReader¶
buildColumnarReader(
partitionedFile: PartitionedFile): PartitionReader[ColumnarBatch]
PartitionReader (of ColumnarBatchs)
See:
Used when:
FilePartitionReaderFactoryis requested to create a columnar reader
Options¶
options: FileSourceOptions
See:
Used when:
FilePartitionReaderFactoryis requested to create a reader and columnar reader
Implementations¶
AvroPartitionReaderFactoryCSVPartitionReaderFactoryJsonPartitionReaderFactoryOrcPartitionReaderFactory- ParquetPartitionReaderFactory
TextPartitionReaderFactory
Creating PartitionReader¶
Signature
createReader(
partition: InputPartition): PartitionReader[InternalRow]
createReader is part of the PartitionReaderFactory abstraction.
createReader...FIXME
Creating Columnar PartitionReader¶
Signature
createColumnarReader(
partition: InputPartition): PartitionReader[ColumnarBatch]
createColumnarReader is part of the PartitionReaderFactory abstraction.
createColumnarReader makes sure that the given InputPartition is a FilePartition (or throws an AssertionError).
createColumnarReader creates a new columnar PartitionReader for every PartitionedFile (of the FilePartition).
In the end, createColumnarReader creates a FilePartitionReader for the files.