DeltaSourceSnapshot
= DeltaSourceSnapshot
[[SnapshotIterator]][[StateCache]] DeltaSourceSnapshot
is a <
DeltaSourceSnapshot
is <DeltaSource
is requested for the <
[[version]] When <DeltaSourceSnapshot
requests the <
== [[creating-instance]] Creating DeltaSourceSnapshot Instance
DeltaSourceSnapshot
takes the following to be created:
- [[spark]]
SparkSession
- [[snapshot]] <
> - [[filters]] Filter expressions (
Seq[Expression]
)
== [[initialFiles]] Initial Files (Indexed AddFiles) -- initialFiles
Method
[source, scala]¶
initialFiles: Dataset[IndexedFile]¶
initialFiles
requests the <Dataset[AddFile]
) and sorts them by <
initialFiles
zips the <RDD.zipWithIndex
operator), adds two new columns with the <isLast
as false
, and finally creates a Dataset[IndexedFile]
.
In the end, initialFiles
<
Delta Source Snapshot #[version] - [redactedPath]
NOTE: initialFiles
is used exclusively when SnapshotIterator
is requested for a <