Skip to content

DeltaTableUtils Utility


  session: SparkSession,
  path: String): (String, Option[DeltaTimeTravelSpec])


extractIfPathContainsTimeTravel is used when:

resolveTimeTravelVersion Utility

  conf: SQLConf,
  deltaLog: DeltaLog,
  tt: DeltaTimeTravelSpec): (Long, String)


resolveTimeTravelVersion is used when:

== [[isDeltaTable]] isDeltaTable Utility

[source, scala]

isDeltaTable( spark: SparkSession, path: Path): Boolean

isDeltaTable tries to <> for the given path and returns whether it was successful or not.

NOTE: isDeltaTable is used when DeltaTable utility is used to <> or <>.

== [[findDeltaTableRoot]] findDeltaTableRoot Utility

[source, scala]

findDeltaTableRoot( spark: SparkSession, path: Path): Option[Path]

findDeltaTableRoot traverses the Hadoop DFS-compliant path upwards (to the root directory of the file system) until _delta_log or _samples directories are found, or the root directory is reached.

For _delta_log or _samples directories, findDeltaTableRoot returns the parent directory.


findDeltaTableRoot is used when:

  • <> is executed

  • DeltaTableUtils utility is used to <>

* DeltaDataSource is requested to <>

== [[splitMetadataAndDataPredicates]] splitMetadataAndDataPredicates Utility

[source, scala]

splitMetadataAndDataPredicates( condition: Expression, partitionColumns: Seq[String], spark: SparkSession): (Seq[Expression], Seq[Expression])



splitMetadataAndDataPredicates is used when:

  • PartitionFiltering is requested to[filesForScan]

*[DeleteCommand] and[UpdateCommand] are executed

== [[isPredicatePartitionColumnsOnly]] isPredicatePartitionColumnsOnly Utility

[source, scala]

isPredicatePartitionColumnsOnly( condition: Expression, partitionColumns: Seq[String], spark: SparkSession): Boolean



isPredicatePartitionColumnsOnly is used when...FIXME

== [[combineWithCatalogMetadata]] combineWithCatalogMetadata Utility

[source, scala]

combineWithCatalogMetadata( sparkSession: SparkSession, table: CatalogTable): CatalogTable


NOTE: combineWithCatalogMetadata seems unused.

Last update: 2020-12-07