Skip to content

Metadata Columns

Spark 3.1.1 (SPARK-31255) introduced support for MetadataColumns for additional metadata of a row.

MetadataColumns can be defined for Tables with SupportsMetadataColumns.

Use DESCRIBE TABLE EXTENDED SQL command to display the metadata columns of a table.

Logical Operators

Logical operators propagate metadata columns using metadataOutput.

ExposesMetadataColumns logical operators can generate metadata columns.

DataSourceV2Relation

MetadataColumns are disregarded (filtered out) from the metadataOutput in DataSourceV2Relation leaf logical operator when in name-conflict with output columns.

__qualified_access_only

__qualified_access_only special metadata attribute is used as a marker for qualified-access-only restriction.