Scd1MergeStreamingWrite¶

Scd1MergeStreamingWrite is a StreamingFlowExecution with AutoCdcMergeWriteBase for AutoCdcMergeFlows (of Type1 with Table output).

Creating Instance¶

Scd1MergeStreamingWrite takes the following to be created:

• TableIdentifier
• AutoCdcMergeFlow
• DataflowGraph
• PipelineUpdateContext
• Checkpoint Path
• Trigger
• Table
• SQL Configuration

Scd1MergeStreamingWrite is created when:

• FlowPlanner is requested to plan an AutoCdcMergeFlow (of Type1 with Table output)

Start Streaming Query¶

startStream(): StreamingQuery

startStream gets the source's Change Data Feed. startStream requests this DataflowGraph to resolve this AutoCdcMergeFlow to take the DataFrame.

startStream createAuxiliaryTableIfNotExists.

startStream creates a Scd1ForeachBatchHandler with a Scd1BatchProcessor.

In the end, startStream starts a streaming query with foreachBatch (Spark Structured Streaming) streaming operator. foreachBatch operator executes the Scd1ForeachBatchHandler for every streaming batch.