Skip to content

DeltaCommand

DeltaCommand is a marker interface for commands to work with data in delta tables.

Implementations

parsePartitionPredicates Method

parsePartitionPredicates(
  spark: SparkSession,
  predicate: String): Seq[Expression]

parsePartitionPredicates...FIXME

parsePartitionPredicates is used when...FIXME

verifyPartitionPredicates Method

verifyPartitionPredicates(
  spark: SparkSession,
  partitionColumns: Seq[String],
  predicates: Seq[Expression]): Unit

verifyPartitionPredicates...FIXME

verifyPartitionPredicates is used when...FIXME

generateCandidateFileMap Method

generateCandidateFileMap(
  basePath: Path,
  candidateFiles: Seq[AddFile]): Map[String, AddFile]

generateCandidateFileMap...FIXME

generateCandidateFileMap is used when...FIXME

removeFilesFromPaths Method

removeFilesFromPaths(
  deltaLog: DeltaLog,
  nameToAddFileMap: Map[String, AddFile],
  filesToRewrite: Seq[String],
  operationTimestamp: Long): Seq[RemoveFile]

removeFilesFromPaths...FIXME

removeFilesFromPaths is used when:

Creating HadoopFsRelation (with TahoeBatchFileIndex)

buildBaseRelation(
  spark: SparkSession,
  txn: OptimisticTransaction,
  actionType: String,
  rootPath: Path,
  inputLeafFiles: Seq[String],
  nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation

buildBaseRelation converts the given inputLeafFiles to AddFiles (with the given rootPath and nameToAddFileMap).

buildBaseRelation creates a TahoeBatchFileIndex for the AddFiles (with the input actionType and rootPath).

In the end, buildBaseRelation creates a HadoopFsRelation (Spark SQL) with the TahoeBatchFileIndex (and the other properties based on the metadata of the given OptimisticTransaction).

buildBaseRelation is used when:

getTouchedFile Method

getTouchedFile(
  basePath: Path,
  filePath: String,
  nameToAddFileMap: Map[String, AddFile]): AddFile

getTouchedFile...FIXME

getTouchedFile is used when:

isCatalogTable Method

isCatalogTable(
  analyzer: Analyzer,
  tableIdent: TableIdentifier): Boolean

isCatalogTable...FIXME

isCatalogTable is used when...FIXME

isPathIdentifier Method

isPathIdentifier(
  tableIdent: TableIdentifier): Boolean

isPathIdentifier...FIXME

isPathIdentifier is used when...FIXME

commitLarge

commitLarge(
  spark: SparkSession,
  txn: OptimisticTransaction,
  actions: Iterator[Action],
  op: DeltaOperations.Operation,
  context: Map[String, String],
  metrics: Map[String, String]): Long

commitLarge...FIXME

commitLarge is used when:

updateAndCheckpoint

updateAndCheckpoint(
  spark: SparkSession,
  deltaLog: DeltaLog,
  commitSize: Int,
  attemptVersion: Long): Unit

updateAndCheckpoint...FIXME


Last update: 2021-06-12
Back to top