Skip to content

ParquetDataSourceV2

ParquetDataSourceV2 is the FileDataSourceV2 of parquet data source.

DataSourceRegister

ParquetDataSourceV2 is registered in META-INF/services/org.apache.spark.sql.sources.DataSourceRegister (for DataSource utility to look up a data source for parquet alias).

Creating Instance

ParquetDataSourceV2 takes no arguments to be created.

ParquetDataSourceV2 is created when:

getTable

getTable(
  options: CaseInsensitiveStringMap): Table
getTable(
  options: CaseInsensitiveStringMap,
  schema: StructType): Table

getTable getPaths from the given options.

getTable getTableName (from the given options and the paths).

getTable getOptionsWithoutPaths.

In the end, getTable creates a ParquetTable.

getTable is part of the FileDataSourceV2 abstraction.

shortName

shortName(): String

shortName is parquet.

shortName is part of the DataSourceRegister abstraction.

fallbackFileFormat

fallbackFileFormat: Class[_ <: FileFormat]

fallbackFileFormat is ParquetFileFormat.

fallbackFileFormat is part of the FileDataSourceV2 abstraction.

Back to top