DeltaTableUtils Utility¶
extractIfPathContainsTimeTravel¶
extractIfPathContainsTimeTravel(
session: SparkSession,
path: String): (String, Option[DeltaTimeTravelSpec])
extractIfPathContainsTimeTravel
...FIXME
extractIfPathContainsTimeTravel
is used when:
DeltaDataSource
is requested to sourceSchema and parsePathIdentifier
resolveTimeTravelVersion Utility¶
resolveTimeTravelVersion(
conf: SQLConf,
deltaLog: DeltaLog,
tt: DeltaTimeTravelSpec): (Long, String)
resolveTimeTravelVersion
...FIXME
resolveTimeTravelVersion
is used when:
DeltaLog
is requested to create a relation (per partition filters and time travel)DeltaTableV2
is requested for a Snapshot
== [[isDeltaTable]] isDeltaTable Utility
[source, scala]¶
isDeltaTable( spark: SparkSession, path: Path): Boolean
isDeltaTable
tries to <
NOTE: isDeltaTable
is used when DeltaTable
utility is used to <
== [[findDeltaTableRoot]] findDeltaTableRoot Utility
[source, scala]¶
findDeltaTableRoot( spark: SparkSession, path: Path): Option[Path]
findDeltaTableRoot
traverses the Hadoop DFS-compliant path upwards (to the root directory of the file system) until _delta_log
or _samples
directories are found, or the root directory is reached.
For _delta_log
or _samples
directories, findDeltaTableRoot
returns the parent directory.
[NOTE]¶
findDeltaTableRoot
is used when:
-
<
> is executed -
DeltaTableUtils
utility is used to <>
* DeltaDataSource
is requested to <>¶
== [[splitMetadataAndDataPredicates]] splitMetadataAndDataPredicates Utility
[source, scala]¶
splitMetadataAndDataPredicates( condition: Expression, partitionColumns: Seq[String], spark: SparkSession): (Seq[Expression], Seq[Expression])
splitMetadataAndDataPredicates
...FIXME
[NOTE]¶
splitMetadataAndDataPredicates
is used when:
PartitionFiltering
is requested to PartitionFiltering.md#filesForScan[filesForScan]
* DeleteCommand.md[DeleteCommand] and UpdateCommand.md[UpdateCommand] are executed¶
== [[isPredicatePartitionColumnsOnly]] isPredicatePartitionColumnsOnly Utility
[source, scala]¶
isPredicatePartitionColumnsOnly( condition: Expression, partitionColumns: Seq[String], spark: SparkSession): Boolean
isPredicatePartitionColumnsOnly
...FIXME
[NOTE]¶
isPredicatePartitionColumnsOnly
is used when...FIXME¶
== [[combineWithCatalogMetadata]] combineWithCatalogMetadata Utility
[source, scala]¶
combineWithCatalogMetadata( sparkSession: SparkSession, table: CatalogTable): CatalogTable
combineWithCatalogMetadata
...FIXME
NOTE: combineWithCatalogMetadata
seems unused.