Skip to content

SupportsMetadata

SupportsMetadata is an abstraction of file-based scan operators that can report custom metadata for formatted explain (e.g., Dataset.explain or EXPLAIN FORMATTED SQL statement).

val query = spark.read.parquet("demo.parquet")
query.explain(mode = "formatted")

== Physical Plan ==
* ColumnarToRow (2)
+- Scan parquet  (1)


(1) Scan parquet
Output [1]: [id#14L]
Batched: true
Location: InMemoryFileIndex [file:/Users/jacek/dev/oss/spark/demo.parquet]
ReadSchema: struct<id:bigint>

(2) ColumnarToRow [codegen id : 1]
Input [1]: [id#14L]

EXPLAIN FORMATTED
SELECT * FROM parquet.`demo.parquet`;

== Physical Plan ==

* ColumnarToRow (2)
+- Scan parquet  (1)

(1) Scan parquet
Output [1]: [id#22L]
Batched: true
Location: InMemoryFileIndex [file:/Users/jacek/dev/oss/spark/demo.parquet]
ReadSchema: struct<id:bigint>

(2) ColumnarToRow [codegen id : 1]
Input [1]: [id#22L]

Contract

Custom Metadata

getMetaData(): Map[String, String]

See:

Used when:

Implementations