DeltaTableUtils Utility¶

DeltaTableUtils comes with the following utilities:

• <> for checking out whether a given directory is part of delta table

• <> for finding the root directory of a delta table

• <>

extractIfPathContainsTimeTravel¶

extractIfPathContainsTimeTravel(
  session: SparkSession,
  path: String): (String, Option[DeltaTimeTravelSpec])

extractIfPathContainsTimeTravel...FIXME

extractIfPathContainsTimeTravel is used when DeltaDataSource is requested to sourceSchema and parsePathIdentifier.

resolveTimeTravelVersion Utility¶

resolveTimeTravelVersion(
  conf: SQLConf,
  deltaLog: DeltaLog,
  tt: DeltaTimeTravelSpec): (Long, String)

resolveTimeTravelVersion...FIXME

resolveTimeTravelVersion is used when:

• DeltaLog is requested to create a relation (per partition filters and time travel)
• DeltaTableV2 is requested for a Snapshot

== [[isDeltaTable]] isDeltaTable Utility

[source, scala]¶

isDeltaTable( spark: SparkSession, path: Path): Boolean

isDeltaTable tries to <> for the given path and returns whether it was successful or not.

NOTE: isDeltaTable is used when DeltaTable utility is used to <> or <>.

== [[findDeltaTableRoot]] findDeltaTableRoot Utility

[source, scala]¶

findDeltaTableRoot( spark: SparkSession, path: Path): Option[Path]

findDeltaTableRoot traverses the Hadoop DFS-compliant path upwards (to the root directory of the file system) until _delta_log or _samples directories are found, or the root directory is reached.

For _delta_log or _samples directories, findDeltaTableRoot returns the parent directory.

[NOTE]¶

findDeltaTableRoot is used when:

• <> is executed

• DeltaTableUtils utility is used to <>

* DeltaDataSource is requested to <>¶

== [[splitMetadataAndDataPredicates]] splitMetadataAndDataPredicates Utility

[source, scala]¶

splitMetadataAndDataPredicates( condition: Expression, partitionColumns: Seq[String], spark: SparkSession): (Seq[Expression], Seq[Expression])

splitMetadataAndDataPredicates...FIXME

[NOTE]¶

splitMetadataAndDataPredicates is used when:

• PartitionFiltering is requested to PartitionFiltering.md#filesForScan[filesForScan]

* DeleteCommand.md[DeleteCommand] and UpdateCommand.md[UpdateCommand] are executed¶

== [[isPredicatePartitionColumnsOnly]] isPredicatePartitionColumnsOnly Utility

[source, scala]¶

isPredicatePartitionColumnsOnly( condition: Expression, partitionColumns: Seq[String], spark: SparkSession): Boolean

isPredicatePartitionColumnsOnly...FIXME

[NOTE]¶

isPredicatePartitionColumnsOnly is used when...FIXME¶

== [[combineWithCatalogMetadata]] combineWithCatalogMetadata Utility

[source, scala]¶

combineWithCatalogMetadata( sparkSession: SparkSession, table: CatalogTable): CatalogTable

combineWithCatalogMetadata...FIXME

NOTE: combineWithCatalogMetadata seems unused.