Spark Connector¶
Spark Connector allows Apache Spark to use Unity Catalog as a metastore.
The following features are supported:
- Read/write (partitioned) external parquet tables
- Read/write (partitioned) external delta tables
- Support the path table syntax of delta tables
- Temporary credentials for secured access to local and S3 tables (parquet tables only, no delta tables yet)
- Namespace
Spark Connector supports Delta Lake, Iceberg and Hudi tables via UniForm.
No Native Iceberg Table Support
There is no native Iceberg table support. It is slated for v0.3 or v0.4 per the Proposed UC Roadmap CY2024Q4.
Spark Connector is UCSingleCatalog and UCProxy.
Spark and Java Compatibility¶
Apache Spark 3.5.3 and Java 11 are used to build Spark Connector module for better Apache Spark interoperability (see this commit).
Namespace Support¶
As of this commit, Unity Catalog supports various namespace-related commands (e.g., SHOW NAMESPACES, DESC NAMESPACE).
See this demo to learn more.