AbstractDeltaCatalog¶

AbstractDeltaCatalog is a DelegatingCatalogExtension.

AbstractDeltaCatalog is a StagingTableCatalog.

AbstractDeltaCatalog is a SupportsPathIdentifier.

Implementations¶

DeltaCatalog
DeltaCatalogV1 (V1 legacy implementation)

Catalog-Managed Delta Tables¶

AbstractDeltaCatalog can create Catalog-Managed Delta Tables when executed with Unity Catalog installed and the table type is MANAGED.

Managed Tables¶

AbstractDeltaCatalog can create a managed delta table when one of the following holds:

it is a catalog table (not based on a path identifier) with no location table propery specified
is_managed_location reserved property is true and Unity Catalog is installed

loadCatalogTable¶

loadCatalogTable(
  ident: Identifier,
  catalogTable: CatalogTable): Table

loadCatalogTable creates a DeltaTableV2.

loadCatalogTable is used when:

DeltaCatalog is requested to load a CatalogTable
AbstractDeltaCatalog is requested to load a delta table

DelegatingCatalogExtension¶

AbstractDeltaCatalog is a DelegatingCatalogExtension (Spark SQL).

createTable¶

DelegatingCatalogExtension

createTable(
  ident: Identifier,
  schema: StructType,
  partitions: Array[Transform],
  properties: util.Map[String, String]): Table

createTable is part of the DelegatingCatalogExtension (Spark SQL) abstraction.

createTable...FIXME

loadTable¶

DelegatingCatalogExtension

loadTable(
  ident: Identifier): Table

loadTable is part of the DelegatingCatalogExtension (Spark SQL) abstraction.

loadTable setVariantBlockingConfigIfUC.

loadTable...FIXME

StagingTableCatalog¶

AbstractDeltaCatalog is a StagingTableCatalog (Spark SQL) that creates a StagedDeltaTableV2 (for delta data source) or a BestEffortStagedTable.

stageCreate¶

StagingTableCatalog

stageCreate(
  ident: Identifier,
  schema: StructType,
  partitions: Array[Transform],
  properties: util.Map[String, String]): StagedTable

stageCreate is part of the StagingTableCatalog (Spark SQL) abstraction.

stageCreate creates a StagedDeltaTableV2 (with TableCreationModes.Create operation) for delta data source only (based on the given properties or spark.sql.sources.default configuration property).

Otherwise, stageCreate creates a BestEffortStagedTable (requesting the parent TableCatalog to create a table).

stageCreateOrReplace¶

StagingTableCatalog

stageCreateOrReplace(
  ident: Identifier,
  schema: StructType,
  partitions: Array[Transform],
  properties: util.Map[String, String]): StagedTable

stageCreateOrReplace is part of the StagingTableCatalog (Spark SQL) abstraction.

stageCreateOrReplace creates a StagedDeltaTableV2 (with TableCreationModes.CreateOrReplace operation) for delta data source only (based on the given properties or spark.sql.sources.default configuration property).

Otherwise, stageCreateOrReplace requests the parent TableCatalog to drop the table first and then creates a BestEffortStagedTable (requesting the parent TableCatalog to create the table).

stageReplace¶

StagingTableCatalog

stageReplace(
  ident: Identifier,
  schema: StructType,
  partitions: Array[Transform],
  properties: util.Map[String, String]): StagedTable

stageReplace is part of the StagingTableCatalog (Spark SQL) abstraction.

stageReplace creates a StagedDeltaTableV2 (with TableCreationModes.Replace operation) for delta data source only (based on the given properties or spark.sql.sources.default configuration property).

Otherwise, stageReplace requests the parent TableCatalog to drop the table first and then creates a BestEffortStagedTable (requesting the parent TableCatalog to create the table).

Unity Catalog Execution Mode¶

isUnityCatalog: Boolean

isUnityCatalog is enabled (true) when the delegate field of this implementation of DelegatingCatalogExtension is a class in io.unitycatalog. package.

In other words, isUnityCatalog is true if the underlying catalog is a Unity Catalog implementation (e.g., io.unitycatalog.spark.UCSingleCatalog).

Lazy Value

isUnityCatalog is a Scala lazy value to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards.

Learn more in the Scala Language Specification.

setVariantBlockingConfigIfUC¶

setVariantBlockingConfigIfUC(): Unit

When in Unity Catalog execution mode, setVariantBlockingConfigIfUC sets spark.databricks.delta.variant.disableVariantTableFeatureForSpark40 configuration property to true.

setVariantBlockingConfigIfUC is used when:

AbstractDeltaCatalog is requested to createTable and loadTable

getTablePropsAndWriteOptions¶

getTablePropsAndWriteOptions(
  properties: JMap[String, String]): (JMap[String, String], Map[String, String])

getTablePropsAndWriteOptions...FIXME

getTablePropsAndWriteOptions is used when:

AbstractDeltaCatalog is requested to createTable
StagedDeltaTableV2 is requested to commitStagedChanges

Creating Delta Table¶

createDeltaTable(
  ident: Identifier,
  schema: StructType,
  partitions: Array[Transform],
  allTableProperties: util.Map[String, String],
  writeOptions: Map[String, String],
  sourceQuery: Option[DataFrame],
  operation: TableCreationModes.CreationMode): Table

createDeltaTable...FIXME

createDeltaTable is used when:

AbstractDeltaCatalog is requested to createTable
StagedDeltaTableV2 is requested to commitStagedChanges

validateClusterBySpec¶

validateClusterBySpec(
  maybeClusterBySpec: Option[ClusterBySpec],
  schema: StructType): Unit

validateClusterBySpec...FIXME