DeltaCommand¶

DeltaCommand is a marker interface for commands to work with data in delta tables.

Implementations¶

• AlterDeltaTableCommand
• ConvertToDeltaCommand
• DeleteCommand
• MergeIntoCommand
• UpdateCommand
• VacuumCommandImpl
• WriteIntoDelta

parsePartitionPredicates Method¶

parsePartitionPredicates(
  spark: SparkSession,
  predicate: String): Seq[Expression]

parsePartitionPredicates...FIXME

parsePartitionPredicates is used when...FIXME

verifyPartitionPredicates Method¶

verifyPartitionPredicates(
  spark: SparkSession,
  partitionColumns: Seq[String],
  predicates: Seq[Expression]): Unit

verifyPartitionPredicates...FIXME

verifyPartitionPredicates is used when...FIXME

generateCandidateFileMap Method¶

generateCandidateFileMap(
  basePath: Path,
  candidateFiles: Seq[AddFile]): Map[String, AddFile]

generateCandidateFileMap...FIXME

generateCandidateFileMap is used when...FIXME

removeFilesFromPaths Method¶

removeFilesFromPaths(
  deltaLog: DeltaLog,
  nameToAddFileMap: Map[String, AddFile],
  filesToRewrite: Seq[String],
  operationTimestamp: Long): Seq[RemoveFile]

removeFilesFromPaths...FIXME

removeFilesFromPaths is used when:

• DeleteCommand and UpdateCommand commands are executed

Creating HadoopFsRelation (with TahoeBatchFileIndex)¶

buildBaseRelation(
  spark: SparkSession,
  txn: OptimisticTransaction,
  actionType: String,
  rootPath: Path,
  inputLeafFiles: Seq[String],
  nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation

buildBaseRelation converts the given inputLeafFiles to AddFiles (with the given rootPath and nameToAddFileMap).

buildBaseRelation creates a TahoeBatchFileIndex for the AddFiles (with the input actionType and rootPath).

In the end, buildBaseRelation creates a HadoopFsRelation (Spark SQL) with the TahoeBatchFileIndex (and the other properties based on the metadata of the given OptimisticTransaction).

buildBaseRelation is used when:

• DeleteCommand and UpdateCommand commands are executed (with delete and update action types, respectively)

getTouchedFile Method¶

getTouchedFile(
  basePath: Path,
  filePath: String,
  nameToAddFileMap: Map[String, AddFile]): AddFile

getTouchedFile...FIXME

getTouchedFile is used when:

• DeltaCommand is requested to removeFilesFromPaths and create a HadoopFsRelation (for DeleteCommand and UpdateCommand commands)

• MergeIntoCommand is executed

isCatalogTable Method¶

isCatalogTable(
  analyzer: Analyzer,
  tableIdent: TableIdentifier): Boolean

isCatalogTable...FIXME

isCatalogTable is used when...FIXME

isPathIdentifier Method¶

isPathIdentifier(
  tableIdent: TableIdentifier): Boolean

isPathIdentifier...FIXME

isPathIdentifier is used when...FIXME

commitLarge¶

commitLarge(
  spark: SparkSession,
  txn: OptimisticTransaction,
  actions: Iterator[Action],
  op: DeltaOperations.Operation,
  context: Map[String, String],
  metrics: Map[String, String]): Long

commitLarge...FIXME

commitLarge is used when:

• ConvertToDeltaCommand command is executed

updateAndCheckpoint¶

updateAndCheckpoint(
  spark: SparkSession,
  deltaLog: DeltaLog,
  commitSize: Int,
  attemptVersion: Long): Unit

updateAndCheckpoint...FIXME