ParquetTable¶

ParquetTable is a FileTable.

Creating Instance¶

ParquetTable takes the following to be created:

• Name
• SparkSession
• Case-insensitive options
• Paths
• User-specified schema
• Fallback FileFormat

ParquetTable is created when:

• ParquetDataSourceV2 is requested for a Table

Format Name¶

formatName: String

formatName is part of the FileTable abstraction.

formatName is the following text:

Parquet

Schema Inference¶

inferSchema(
  files: Seq[FileStatus]): Option[StructType]

inferSchema is part of the FileTable abstraction.

inferSchema infers the schema (with the options and the input Hadoop FileStatuses).

newScanBuilder¶

newScanBuilder(
  options: CaseInsensitiveStringMap): ParquetScanBuilder

newScanBuilder is part of the FileTable abstraction.

newScanBuilder creates a ParquetScanBuilder with the following:

• fileIndex
• schema
• dataSchema
• options

newWriteBuilder¶

newWriteBuilder(
  info: LogicalWriteInfo): WriteBuilder

newWriteBuilder is part of the FileTable abstraction.

newWriteBuilder creates a WriteBuilder that creates a ParquetWrite (when requested to build a Write).

supportsDataType¶

supportsDataType(
  dataType: DataType): Boolean

supportsDataType is part of the FileTable abstraction.

supportsDataType supports all AtomicTypes and the following complex DataTypes with AtomicTypes:

• StructType
• ArrayType
• MapType
• UserDefinedType