Skip to content

Testing Streaming Queries

Spark Structured Streaming comes with the built-in Memory Data Source for writing tests of streaming queries.

The idea is to load data from a memory source, execute data stream transformations (your code), and write the result out to a memory sink (that becomes a queryable temporary view).

Demo

Tip

Review MemorySinkSuite and FileStreamSourceSuite test suites in the source code of Apache Spark.

Learn More