DeltaCommand¶
DeltaCommand is a marker interface for commands to work with data in delta tables.
Implementations¶
- AlterDeltaTableCommand
- ConvertToDeltaCommand
- DeleteCommand
- MergeIntoCommand
- UpdateCommand
- VacuumCommandImpl
- WriteIntoDelta
parsePartitionPredicates Method¶
parsePartitionPredicates(
spark: SparkSession,
predicate: String): Seq[Expression]
parsePartitionPredicates
...FIXME
parsePartitionPredicates
is used when...FIXME
verifyPartitionPredicates Method¶
verifyPartitionPredicates(
spark: SparkSession,
partitionColumns: Seq[String],
predicates: Seq[Expression]): Unit
verifyPartitionPredicates
...FIXME
verifyPartitionPredicates
is used when...FIXME
generateCandidateFileMap Method¶
generateCandidateFileMap(
basePath: Path,
candidateFiles: Seq[AddFile]): Map[String, AddFile]
generateCandidateFileMap
...FIXME
generateCandidateFileMap
is used when...FIXME
removeFilesFromPaths Method¶
removeFilesFromPaths(
deltaLog: DeltaLog,
nameToAddFileMap: Map[String, AddFile],
filesToRewrite: Seq[String],
operationTimestamp: Long): Seq[RemoveFile]
removeFilesFromPaths
...FIXME
removeFilesFromPaths
is used when DeleteCommand and UpdateCommand commands are executed.
Creating HadoopFsRelation (with TahoeBatchFileIndex)¶
buildBaseRelation(
spark: SparkSession,
txn: OptimisticTransaction,
actionType: String,
rootPath: Path,
inputLeafFiles: Seq[String],
nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation
buildBaseRelation
converts the given inputLeafFiles
to AddFiles (with the given rootPath
and nameToAddFileMap
).
buildBaseRelation
creates a TahoeBatchFileIndex for the actionType
, the AddFiles and the rootPath
.
In the end, buildBaseRelation
creates a HadoopFsRelation
with the TahoeBatchFileIndex
(and the other properties based on the metadata of the given OptimisticTransaction).
Note
Learn more on HadoopFsRelation in The Internals of Spark SQL online book.
buildBaseRelation
is used when DeleteCommand and UpdateCommand commands are executed (with delete
and update
action types, respectively).
getTouchedFile Method¶
getTouchedFile(
basePath: Path,
filePath: String,
nameToAddFileMap: Map[String, AddFile]): AddFile
getTouchedFile
...FIXME
getTouchedFile
is used when:
-
DeltaCommand
is requested to removeFilesFromPaths and create a HadoopFsRelation (for DeleteCommand and UpdateCommand commands) -
MergeIntoCommand is executed
isCatalogTable Method¶
isCatalogTable(
analyzer: Analyzer,
tableIdent: TableIdentifier): Boolean
isCatalogTable
...FIXME
isCatalogTable
is used when...FIXME
isPathIdentifier Method¶
isPathIdentifier(
tableIdent: TableIdentifier): Boolean
isPathIdentifier
...FIXME
isPathIdentifier
is used when...FIXME