Metadata Columns¶

Spark 3.1.1 (SPARK-31255) introduced support for MetadataColumns for additional metadata of a row.

MetadataColumns can be defined for Tables with SupportsMetadataColumns.

Use DESCRIBE TABLE EXTENDED SQL command to display the metadata columns of a table.

__metadata_col¶

__metadata_col is used when:

• MetadataAttribute is created and destructured
• FileSourceMetadataAttribute is created and requested to removeInternalMetadata
• FileSourceConstantMetadataAttribute is created
• FileSourceGeneratedMetadataAttribute is created
• MetadataColumnHelper is requested to isMetadataCol and markAsQualifiedAccessOnly
• MetadataColumnsHelper is requested to asStruct

Logical Operators¶

Logical operators propagate metadata columns using metadataOutput.

ExposesMetadataColumns logical operators can generate metadata columns.

DataSourceV2Relation¶

MetadataColumns are disregarded (filtered out) from the metadataOutput in DataSourceV2Relation leaf logical operator when in name-conflict with output columns.