DeltaSink

DeltaSink is the sink of delta data source for streaming queries in Spark Structured Streaming.

DeltaSink is created exclusively when DeltaDataSource is requested for a streaming sink (Structured Streaming).

DeltaSink uses the following text representation (with the path):

DeltaSink[path]

Creating Instance

DeltaSink takes the following to be created:

deltaLog Internal Property

deltaLog: DeltaLog

deltaLog is a DeltaLog that is created for the delta table when DeltaSink is created (when DeltaDataSource is requested for a streaming sink).

deltaLog is used exclusively when DeltaSink is requested to add a streaming micro-batch.

Adding Streaming Micro-Batch

addBatch(
  batchId: Long,
  data: DataFrame): Unit
addBatch is part of the Sink contract (in Spark Structured Streaming) to add a batch of data to the sink.

addBatch requests the DeltaLog to start a new transaction.

addBatch…​FIXME

In the end, addBatch requests the OptimisticTransaction to commit.