DeltaCommand

DeltaCommand is the abstraction of Delta commands that can FIXME.

Table 1. DeltaCommands (Direct Implementations and Extensions Only)
DeltaCommand Description

WriteIntoDelta

VacuumCommandImpl

UpdateCommand

MergeIntoCommand

ConvertToDeltaCommandBase

DeleteCommand

parsePartitionPredicates Method

parsePartitionPredicates(
  spark: SparkSession,
  predicate: String): Seq[Expression]

parsePartitionPredicates…​FIXME

parsePartitionPredicates is used when…​FIXME

verifyPartitionPredicates Method

verifyPartitionPredicates(
  spark: SparkSession,
  partitionColumns: Seq[String],
  predicates: Seq[Expression]): Unit

verifyPartitionPredicates…​FIXME

verifyPartitionPredicates is used when…​FIXME

generateCandidateFileMap Method

generateCandidateFileMap(
  basePath: Path,
  candidateFiles: Seq[AddFile]): Map[String, AddFile]

generateCandidateFileMap…​FIXME

generateCandidateFileMap is used when…​FIXME

removeFilesFromPaths Method

removeFilesFromPaths(
  deltaLog: DeltaLog,
  nameToAddFileMap: Map[String, AddFile],
  filesToRewrite: Seq[String],
  operationTimestamp: Long): Seq[RemoveFile]

removeFilesFromPaths…​FIXME

removeFilesFromPaths is used when DeleteCommand and UpdateCommand are executed.

Creating HadoopFsRelation (With TahoeBatchFileIndex) — buildBaseRelation Method

buildBaseRelation(
  spark: SparkSession,
  txn: OptimisticTransaction,
  actionType: String,
  rootPath: Path,
  inputLeafFiles: Seq[String],
  nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation

buildBaseRelation converts the given inputLeafFiles to AddFiles (with the given rootPath and nameToAddFileMap).

buildBaseRelation creates a TahoeBatchFileIndex for the actionType, the AddFiles and the rootPath.

In the end, buildBaseRelation creates a HadoopFsRelation with the TahoeBatchFileIndex (and the other properties based on the metadata of the given OptimisticTransaction).

buildBaseRelation is used when DeleteCommand and UpdateCommand are executed (with delete and update action types, respectively).

getTouchedFile Method

getTouchedFile(
  basePath: Path,
  filePath: String,
  nameToAddFileMap: Map[String, AddFile]): AddFile

getTouchedFile…​FIXME

getTouchedFile is used when: