MicroBatchStream¶

MicroBatchStream is an extension of the SparkDataStream abstraction for data streams for Micro-Batch Stream Processing.

Contract¶

Creating PartitionReaderFactory¶

PartitionReaderFactory createReaderFactory()

PartitionReaderFactory (Spark SQL)

Used when:

• MicroBatchScanExec physical operator is requested for a PartitionReaderFactory

Latest Offset¶

Offset latestOffset()

Latest Offset

Used when:

• MicroBatchExecution is requested to constructing or skipping next streaming micro-batch

Input Partitions¶

InputPartition[] planInputPartitions(
  Offset start,
  Offset end)

InputPartitions (Spark SQL)

Used when:

• MicroBatchScanExec physical operator is requested for input partitions

Implementations¶

• AvailableNowMicroBatchStreamWrapper
• KafkaMicroBatchStream
• MemoryStream
• RatePerMicroBatchStream
• RateStreamMicroBatchStream
• TextSocketMicroBatchStream