MetadataBuilder¶

MetadataBuilder is used to describe how to create a Metadata (using Fluent API).

import org.apache.spark.sql.types.MetadataBuilder

val newMetadata = new MetadataBuilder()
  .withMetadata(attr.metadata)
  .putNull("__is_duplicate")
  .build()

Creating Metadata¶

build(): Metadata

build creates a Metadata (for the internal map).

withMetadata¶

withMetadata(
  metadata: Metadata): this.type

withMetadata adds the map of the given Metadata to the internal map.

In the end, withMetadata returns this MetadataBuilder.

withMetadata is used when:

• FileSourceConstantMetadataStructField is requested for metadata
• FileSourceGeneratedMetadataStructField is requested for metadata
• FileSourceMetadataAttribute is requested for metadata and removeInternalMetadata
• MetadataColumnHelper is requested to markAsAllowAnyAccess and markAsQualifiedAccessOnly
• ResolveDefaultColumns logical resolution rule is executed (to constantFoldCurrentDefaultsToExistDefaults)
• StructField is requested to remove or update the CURRENT_DEFAULT metadata attribute, and update EXISTS_DEFAULT metadata attribute
• others