DeltaSink takes the following to be created:
- Hadoop Path of the delta table (to write data to as configured by the path option)
- Names of the partition columns
OutputMode(Spark Structured Streaming)
DeltaSink is created when:
DeltaDataSourceis requested for a streaming sink
deltaLog is used when:
- DeltaSink is requested to add a streaming micro-batch
Adding Streaming Micro-Batch¶
addBatch( batchId: Long, data: DataFrame): Unit
addBatch is part of the
Sink (Spark Structured Streaming) abstraction.
In the end,
addBatch requests the
OptimisticTransaction to commit.
DeltaSink uses the following text representation (with the path):
DeltaSink is an ImplicitMetadataOperation.