Skip to content

DeltaTableUtils Utility

DeltaTableUtils comes with the following utilities:

  • <> for checking out whether a given directory is part of delta table

  • <> for finding the root directory of a delta table

  • <>

extractIfPathContainsTimeTravel

extractIfPathContainsTimeTravel(
  session: SparkSession,
  path: String): (String, Option[DeltaTimeTravelSpec])

extractIfPathContainsTimeTravel...FIXME

extractIfPathContainsTimeTravel is used when DeltaDataSource is requested to sourceSchema and parsePathIdentifier.

resolveTimeTravelVersion Utility

resolveTimeTravelVersion(
  conf: SQLConf,
  deltaLog: DeltaLog,
  tt: DeltaTimeTravelSpec): (Long, String)

resolveTimeTravelVersion...FIXME

resolveTimeTravelVersion is used when:

== [[isDeltaTable]] isDeltaTable Utility

[source, scala]

isDeltaTable( spark: SparkSession, path: Path): Boolean


isDeltaTable tries to <> for the given path and returns whether it was successful or not.

NOTE: isDeltaTable is used when DeltaTable utility is used to <> or <>.

== [[findDeltaTableRoot]] findDeltaTableRoot Utility

[source, scala]

findDeltaTableRoot( spark: SparkSession, path: Path): Option[Path]


findDeltaTableRoot traverses the Hadoop DFS-compliant path upwards (to the root directory of the file system) until _delta_log or _samples directories are found, or the root directory is reached.

For _delta_log or _samples directories, findDeltaTableRoot returns the parent directory.

[NOTE]

findDeltaTableRoot is used when:

  • <> is executed

  • DeltaTableUtils utility is used to <>

* DeltaDataSource is requested to <>

== [[splitMetadataAndDataPredicates]] splitMetadataAndDataPredicates Utility

[source, scala]

splitMetadataAndDataPredicates( condition: Expression, partitionColumns: Seq[String], spark: SparkSession): (Seq[Expression], Seq[Expression])


splitMetadataAndDataPredicates...FIXME

[NOTE]

splitMetadataAndDataPredicates is used when:

  • PartitionFiltering is requested to PartitionFiltering.md#filesForScan[filesForScan]

* DeleteCommand.md[DeleteCommand] and UpdateCommand.md[UpdateCommand] are executed

== [[isPredicatePartitionColumnsOnly]] isPredicatePartitionColumnsOnly Utility

[source, scala]

isPredicatePartitionColumnsOnly( condition: Expression, partitionColumns: Seq[String], spark: SparkSession): Boolean


isPredicatePartitionColumnsOnly...FIXME

[NOTE]

isPredicatePartitionColumnsOnly is used when...FIXME

== [[combineWithCatalogMetadata]] combineWithCatalogMetadata Utility

[source, scala]

combineWithCatalogMetadata( sparkSession: SparkSession, table: CatalogTable): CatalogTable


combineWithCatalogMetadata...FIXME

NOTE: combineWithCatalogMetadata seems unused.


Last update: 2020-10-13