SupportsMetadata¶
SupportsMetadata is an abstraction of file-based scan operators that can report custom metadata for formatted explain (e.g., Dataset.explain or EXPLAIN FORMATTED SQL statement).
val query = spark.read.parquet("demo.parquet")
query.explain(mode = "formatted")
== Physical Plan ==
* ColumnarToRow (2)
+- Scan parquet (1)
(1) Scan parquet
Output [1]: [id#14L]
Batched: true
Location: InMemoryFileIndex [file:/Users/jacek/dev/oss/spark/demo.parquet]
ReadSchema: struct<id:bigint>
(2) ColumnarToRow [codegen id : 1]
Input [1]: [id#14L]
EXPLAIN FORMATTED
SELECT * FROM parquet.`demo.parquet`;
== Physical Plan ==
* ColumnarToRow (2)
+- Scan parquet (1)
(1) Scan parquet
Output [1]: [id#22L]
Batched: true
Location: InMemoryFileIndex [file:/Users/jacek/dev/oss/spark/demo.parquet]
ReadSchema: struct<id:bigint>
(2) ColumnarToRow [codegen id : 1]
Input [1]: [id#22L]
Contract¶
Custom Metadata¶
getMetaData(): Map[String, String]
See:
Used when:
- DataSourceV2ScanExecBase physical operator is executed (and verboseStringWithOperatorId)