AddFile takes the following to be created:
- Partition values (
- Size (in bytes)
- Modification time
- File Statistics
AddFile is created when:
AddFile can be given a JSON-encoded file statistics when created.
The statistics are undefined (
null) by default.
The statistics can be defined when:
TransactionalWriteis requested to write data out (and spark.databricks.delta.stats.collect configuration property is enabled)
StatisticsCollectionutility is used to recompute statistics for a delta table (that seems unused though)
Converting to SingleAction¶
wrap is part of the Action abstraction.
wrap creates a new SingleAction with the
add field set to this
Converting to RemoveFile with Defaults¶
remove is used when:
- MergeIntoCommand is executed
WriteIntoDeltais requested to write (with
DeltaSinkis requested to add a streaming micro-batch (with
Converting to RemoveFile¶
removeWithTimestamp( timestamp: Long = System.currentTimeMillis(), dataChange: Boolean = true): RemoveFile
removeWithTimestamp is used when: