DeltaCommand¶

DeltaCommand is a marker interface for commands to work with data in delta tables.

Implementations¶

• AlterDeltaTableCommand
• ConvertToDeltaCommand
• DeleteCommand
• MergeIntoCommand
• UpdateCommand
• VacuumCommandImpl
• WriteIntoDelta

parsePartitionPredicates Method¶

parsePartitionPredicates(
  spark: SparkSession,
  predicate: String): Seq[Expression]

parsePartitionPredicates...FIXME

parsePartitionPredicates is used when...FIXME

verifyPartitionPredicates Method¶

verifyPartitionPredicates(
  spark: SparkSession,
  partitionColumns: Seq[String],
  predicates: Seq[Expression]): Unit

verifyPartitionPredicates...FIXME

verifyPartitionPredicates is used when...FIXME

generateCandidateFileMap Method¶

generateCandidateFileMap(
  basePath: Path,
  candidateFiles: Seq[AddFile]): Map[String, AddFile]

generateCandidateFileMap...FIXME

generateCandidateFileMap is used when...FIXME

removeFilesFromPaths Method¶

removeFilesFromPaths(
  deltaLog: DeltaLog,
  nameToAddFileMap: Map[String, AddFile],
  filesToRewrite: Seq[String],
  operationTimestamp: Long): Seq[RemoveFile]

removeFilesFromPaths...FIXME

removeFilesFromPaths is used when DeleteCommand and UpdateCommand commands are executed.

Creating HadoopFsRelation (with TahoeBatchFileIndex)¶

buildBaseRelation(
  spark: SparkSession,
  txn: OptimisticTransaction,
  actionType: String,
  rootPath: Path,
  inputLeafFiles: Seq[String],
  nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation

buildBaseRelation converts the given inputLeafFiles to AddFiles (with the given rootPath and nameToAddFileMap).

buildBaseRelation creates a TahoeBatchFileIndex for the actionType, the AddFiles and the rootPath.

In the end, buildBaseRelation creates a HadoopFsRelation with the TahoeBatchFileIndex (and the other properties based on the metadata of the given OptimisticTransaction).

buildBaseRelation is used when DeleteCommand and UpdateCommand commands are executed (with delete and update action types, respectively).

getTouchedFile Method¶

getTouchedFile(
  basePath: Path,
  filePath: String,
  nameToAddFileMap: Map[String, AddFile]): AddFile

getTouchedFile...FIXME

getTouchedFile is used when:

• DeltaCommand is requested to removeFilesFromPaths and create a HadoopFsRelation (for DeleteCommand and UpdateCommand commands)

• MergeIntoCommand is executed

isCatalogTable Method¶

isCatalogTable(
  analyzer: Analyzer,
  tableIdent: TableIdentifier): Boolean

isCatalogTable...FIXME

isCatalogTable is used when...FIXME

isPathIdentifier Method¶

isPathIdentifier(
  tableIdent: TableIdentifier): Boolean

isPathIdentifier...FIXME

isPathIdentifier is used when...FIXME

commitLarge¶

commitLarge(
  spark: SparkSession,
  txn: OptimisticTransaction,
  actions: Iterator[Action],
  op: DeltaOperations.Operation,
  context: Map[String, String],
  metrics: Map[String, String]): Long

commitLarge...FIXME

commitLarge is used when:

• ConvertToDeltaCommand command is executed