Metadata Columns¶

Spark 3.1.1 (SPARK-31255) introduced support for MetadataColumns for additional metadata of a row.

MetadataColumns can be defined for Tables with SupportsMetadataColumns.

Use DESCRIBE TABLE EXTENDED SQL command to display the metadata columns of a table.

Logical Operators¶

Logical operators propagate metadata columns using metadataOutput.

ExposesMetadataColumns logical operators can generate metadata columns.

DataSourceV2Relation¶

MetadataColumns are disregarded (filtered out) from the metadataOutput in DataSourceV2Relation leaf logical operator when in name-conflict with output columns.