Real-Time Mode¶
Real-Time Mode (RTM) is a new execution model designed to lower end-to-end data processing latency (to the order of 100 milliseconds).
Real-Time Mode lets streaming queries running in real-time mode for continuous, sub-second latency processing.
Stateless tasks can hit single-digit millisecond latency.
The goal of Real-Time Mode is to enable Apache Spark to power real-time applications (like instant anomaly alerts or live personalization) that today cannot meet their latency requirements with Spark's current streaming engine.
Real-Time Mode is not supported with Asynchronous Progress Tracking.
Adaptive Query Execution is not supported in Real-time Mode.
MicroBatchExecution stream execution engine reports new data available constantly.
Real-Time Mode is supported by streaming sources with SupportsRealTimeMode.
Real-Time Mode does not support Sinks due to API limitations (i.e., no writing outputs row by row).
Real-Time Mode supports Update output mode only (that is enforced by UnsupportedOperationChecker).
Apache Spark 4.1.0
Real-Time Mode was introduced in Apache Spark 4.1.0.
Continuous Mode
The experimental Continuous Mode shares the same goals but only supports limited queries.