Skip to content

DeltaCommand

DeltaCommand is a marker interface for commands to work with data in delta tables.

Implementations

parsePartitionPredicates Method

parsePartitionPredicates(
  spark: SparkSession,
  predicate: String): Seq[Expression]

parsePartitionPredicates...FIXME

parsePartitionPredicates is used when...FIXME

verifyPartitionPredicates Method

verifyPartitionPredicates(
  spark: SparkSession,
  partitionColumns: Seq[String],
  predicates: Seq[Expression]): Unit

verifyPartitionPredicates...FIXME

verifyPartitionPredicates is used when...FIXME

generateCandidateFileMap Method

generateCandidateFileMap(
  basePath: Path,
  candidateFiles: Seq[AddFile]): Map[String, AddFile]

generateCandidateFileMap...FIXME

generateCandidateFileMap is used when...FIXME

removeFilesFromPaths Method

removeFilesFromPaths(
  deltaLog: DeltaLog,
  nameToAddFileMap: Map[String, AddFile],
  filesToRewrite: Seq[String],
  operationTimestamp: Long): Seq[RemoveFile]

removeFilesFromPaths...FIXME

removeFilesFromPaths is used when DeleteCommand and UpdateCommand commands are executed.

Creating HadoopFsRelation (with TahoeBatchFileIndex)

buildBaseRelation(
  spark: SparkSession,
  txn: OptimisticTransaction,
  actionType: String,
  rootPath: Path,
  inputLeafFiles: Seq[String],
  nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation

buildBaseRelation converts the given inputLeafFiles to AddFiles (with the given rootPath and nameToAddFileMap).

buildBaseRelation creates a TahoeBatchFileIndex for the actionType, the AddFiles and the rootPath.

In the end, buildBaseRelation creates a HadoopFsRelation with the TahoeBatchFileIndex (and the other properties based on the metadata of the given OptimisticTransaction).

Note

Learn more on HadoopFsRelation in The Internals of Spark SQL online book.

buildBaseRelation is used when DeleteCommand and UpdateCommand commands are executed (with delete and update action types, respectively).

getTouchedFile Method

getTouchedFile(
  basePath: Path,
  filePath: String,
  nameToAddFileMap: Map[String, AddFile]): AddFile

getTouchedFile...FIXME

getTouchedFile is used when:

isCatalogTable Method

isCatalogTable(
  analyzer: Analyzer,
  tableIdent: TableIdentifier): Boolean

isCatalogTable...FIXME

isCatalogTable is used when...FIXME

isPathIdentifier Method

isPathIdentifier(
  tableIdent: TableIdentifier): Boolean

isPathIdentifier...FIXME

isPathIdentifier is used when...FIXME


Last update: 2020-10-05