Skip to content

Metadata Columns

Spark 3.1.1 (SPARK-31255) introduced support for MetadataColumns for additional metadata of a row.

MetadataColumns can be defined for Tables with SupportsMetadataColumns.

Use DESCRIBE TABLE EXTENDED SQL command to display the metadata columns of a table.

__metadata_col

__metadata_col is used when:

Logical Operators

Logical operators propagate metadata columns using metadataOutput.

ExposesMetadataColumns logical operators can generate metadata columns.

DataSourceV2Relation

MetadataColumns are disregarded (filtered out) from the metadataOutput in DataSourceV2Relation leaf logical operator when in name-conflict with output columns.