DeltaCommand¶
DeltaCommand is a marker interface for commands to work with data in delta tables.
Implementations¶
- AlterDeltaTableCommand
- ConvertToDeltaCommand
- DeleteCommand
- MergeIntoCommand
- UpdateCommand
- VacuumCommandImpl
- WriteIntoDelta
parsePartitionPredicates Method¶
parsePartitionPredicates(
spark: SparkSession,
predicate: String): Seq[Expression]
parsePartitionPredicates
...FIXME
parsePartitionPredicates
is used when...FIXME
verifyPartitionPredicates Method¶
verifyPartitionPredicates(
spark: SparkSession,
partitionColumns: Seq[String],
predicates: Seq[Expression]): Unit
verifyPartitionPredicates
...FIXME
verifyPartitionPredicates
is used when...FIXME
generateCandidateFileMap Method¶
generateCandidateFileMap(
basePath: Path,
candidateFiles: Seq[AddFile]): Map[String, AddFile]
generateCandidateFileMap
...FIXME
generateCandidateFileMap
is used when...FIXME
removeFilesFromPaths Method¶
removeFilesFromPaths(
deltaLog: DeltaLog,
nameToAddFileMap: Map[String, AddFile],
filesToRewrite: Seq[String],
operationTimestamp: Long): Seq[RemoveFile]
removeFilesFromPaths
...FIXME
removeFilesFromPaths
is used when:
- DeleteCommand and UpdateCommand commands are executed
Creating HadoopFsRelation (with TahoeBatchFileIndex)¶
buildBaseRelation(
spark: SparkSession,
txn: OptimisticTransaction,
actionType: String,
rootPath: Path,
inputLeafFiles: Seq[String],
nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation
buildBaseRelation
converts the given inputLeafFiles
to AddFiles (with the given rootPath
and nameToAddFileMap
).
buildBaseRelation
creates a TahoeBatchFileIndex for the AddFile
s (with the input actionType
and rootPath
).
In the end, buildBaseRelation
creates a HadoopFsRelation
(Spark SQL) with the TahoeBatchFileIndex
(and the other properties based on the metadata of the given OptimisticTransaction).
buildBaseRelation
is used when:
- DeleteCommand and UpdateCommand commands are executed (with
delete
andupdate
action types, respectively)
getTouchedFile Method¶
getTouchedFile(
basePath: Path,
filePath: String,
nameToAddFileMap: Map[String, AddFile]): AddFile
getTouchedFile
...FIXME
getTouchedFile
is used when:
-
DeltaCommand
is requested to removeFilesFromPaths and create a HadoopFsRelation (for DeleteCommand and UpdateCommand commands) -
MergeIntoCommand is executed
isCatalogTable Method¶
isCatalogTable(
analyzer: Analyzer,
tableIdent: TableIdentifier): Boolean
isCatalogTable
...FIXME
isCatalogTable
is used when...FIXME
isPathIdentifier Method¶
isPathIdentifier(
tableIdent: TableIdentifier): Boolean
isPathIdentifier
...FIXME
isPathIdentifier
is used when...FIXME
commitLarge¶
commitLarge(
spark: SparkSession,
txn: OptimisticTransaction,
actions: Iterator[Action],
op: DeltaOperations.Operation,
context: Map[String, String],
metrics: Map[String, String]): Long
commitLarge
...FIXME
commitLarge
is used when:
- ConvertToDeltaCommand command is executed
updateAndCheckpoint¶
updateAndCheckpoint(
spark: SparkSession,
deltaLog: DeltaLog,
commitSize: Int,
attemptVersion: Long): Unit
updateAndCheckpoint
...FIXME