Skip to content

CatalogManager

Creating Instance

CatalogManager takes the following to be created:

CatalogManager is created when:

Default Session Catalog

defaultSessionCatalog: CatalogPlugin

CatalogManager is given a CatalogPlugin when created for the default session catalog (which is a V2SessionCatalog).

defaultSessionCatalog is used as the delegate catalog when CatalogManager is requested to load a V2SessionCatalog.

defaultSessionCatalog is used as the fallback catalog when CatalogManager is requested to load a custom V2CatalogPlugin.

spark_catalog Default Catalog Name

CatalogManager defines spark_catalog as the name of the default catalog (V2SessionCatalog).

spark_catalog is used as the default value of spark.sql.defaultCatalog configuration property.

Current Catalog Name

_currentCatalogName: Option[String]

_currentCatalogName is the name of the current catalog and is undefined by default (None).

_currentCatalogName can be changed using setCurrentCatalog.

Current CatalogPlugin

currentCatalog: CatalogPlugin

currentCatalog uses the current CatalogPlugin if defined or falls back on spark.sql.defaultCatalog configuration property.

currentCatalog is used when:

Current Namespace

currentNamespace: Array[String]

currentNamespace...FIXME

currentNamespace is used when:

  • ResolveNamespace analysis rule is executed
  • GetCurrentDatabase analysis rule is executed
  • CatalogAndIdentifier extractor utility is requested to unapply
  • ViewHelper utility is requested to generateViewProperties

Setting Current Namespace

setCurrentNamespace(
  namespace: Array[String]): Unit

setCurrentNamespace...FIXME

setCurrentNamespace is used when:

  • SetCatalogAndNamespaceExec physical command is executed

Changing Current Catalog Name

setCurrentCatalog(
  catalogName: String): Unit

setCurrentCatalog checks out whether the given catalog name is different from the currentCatalog's.

Only if the names are different, setCurrentCatalog makes it _currentCatalogName and "resets" _currentNamespace (None). In the end, setCurrentCatalog requests the SessionCatalog to setCurrentDatabase as default.

setCurrentCatalog is used when:

Finding CatalogPlugin by Name

catalog(
  name: String): CatalogPlugin

catalog returns the v2 session catalog when the given name is spark_catalog.

Otherwise, catalog looks up the name in catalogs internal registry. When not found, catalog tries to load a CatalogPlugin by name (and registers it in catalogs internal registry).

catalog is used when:

isCatalogRegistered

isCatalogRegistered(
  name: String): Boolean

isCatalogRegistered...FIXME

isCatalogRegistered is used when Analyzer is requested to expandRelationName.

v2SessionCatalog

v2SessionCatalog: CatalogPlugin

v2SessionCatalog...FIXME

v2SessionCatalog is used when:

  • CatalogManager is requested to look up a CatalogPlugin by name
  • CatalogV2Util is requested to getTableProviderCatalog
  • CatalogAndIdentifier utility is requested to extract a CatalogPlugin and an identifier from a multi-part name (unapply)

loadV2SessionCatalog

loadV2SessionCatalog(): CatalogPlugin

loadV2SessionCatalog loads the default CatalogPlugin that handles spark_catalog catalog.

spark.sql.catalog.spark_catalog

spark.sql.catalog.spark_catalog configuration property is used to specify the fully-qualified class name of the session CatalogPlugin.

Only if the CatalogPlugin is a CatalogExtension, loadV2SessionCatalog assigns this default session CatalogPlugin to be the delegate CatalogPlugin.

loadV2SessionCatalog is used when: