DeltaTableV2¶
DeltaTableV2
is a logical representation of a writable Delta table.
DeltaTableV2 by Spark SQL 3.0.0
Using the abstractions introduced in Spark SQL 3.0.0, DeltaTableV2
is a Table that SupportsWrite.
Creating Instance¶
DeltaTableV2
takes the following to be created:
-
SparkSession
- Hadoop Path
- Optional Catalog Metadata (
Option[CatalogTable]
) - Optional Table ID (
Option[String]
) - Optional DeltaTimeTravelSpec
DeltaTableV2
is created when:
DeltaCatalog
is requested to load a tableDeltaDataSource
is requested to load a table or create a table relation
DeltaTimeTravelSpec¶
DeltaTableV2
may be given a DeltaTimeTravelSpec to be created.
DeltaTimeTravelSpec
is assumed not to be defined.
DeltaTableV2
is given a DeltaTimeTravelSpec
only when DeltaDataSource
is requested to create a BaseRelation.
DeltaTimeTravelSpec
is used for timeTravelSpec.
Snapshot¶
snapshot: Snapshot
DeltaTableV2
has a Snapshot. In other words, DeltaTableV2
represents a Delta table at a specific version.
Scala lazy value
snapshot
is a Scala lazy value and is initialized once when first accessed. Once computed it stays unchanged.
DeltaTableV2
uses the DeltaLog to load it at a given version (based on the optional timeTravelSpec) or update to the latest version.
snapshot
is used when DeltaTableV2
is requested for the schema, partitioning and properties.
DeltaTimeTravelSpec¶
timeTravelSpec: Option[DeltaTimeTravelSpec]
DeltaTableV2
may have a DeltaTimeTravelSpec specified that is either given or timeTravelByPath.
timeTravelSpec
throws an AnalysisException
when timeTravelOpt and timeTravelByPath are both defined:
Cannot specify time travel in multiple formats.
timeTravelSpec
is used when DeltaTableV2
is requested for a Snapshot and BaseRelation.
DeltaTimeTravelSpec by Path¶
timeTravelByPath: Option[DeltaTimeTravelSpec]
Scala lazy value
timeTravelByPath
is a Scala lazy value and is initialized once when first accessed. Once computed it stays unchanged.
timeTravelByPath
is undefined when CatalogTable is defined.
With no CatalogTable defined, DeltaTableV2
parses the given Path for the timeTravelByPath
(that resolvePath under the covers).
Converting to Insertable HadoopFsRelation¶
toBaseRelation: BaseRelation
toBaseRelation
verifyAndCreatePartitionFilters for the Path, the current Snapshot and partitionFilters.
In the end, toBaseRelation
requests the DeltaLog for an insertable HadoopFsRelation.
toBaseRelation
is used when:
DeltaDataSource
is requested to createRelationDeltaRelation
utility is used tofromV2Relation