Testing Streaming Queries¶

Spark Structured Streaming comes with the built-in Memory Data Source for writing tests of streaming queries.

The idea is to load data from a memory source, execute data stream transformations (your code), and write the result out to a memory sink (that becomes a queryable temporary view).

Demo¶

Learn More¶

• How to perform Unit testing on Spark Structured Streaming?
• Unit Testing Spark Structured Streaming Application using Memory Stream
• Integration tests and Structured Streaming
• Writing Unit Test for Apache Spark using Memory Streams