ContinuousReader — Data Source Readers in Continuous Stream Processing¶

ContinuousReader is the <> of Spark SQL's DataSourceReader abstraction for <> in Continuous Stream Processing.

ContinuousReader is part of the novel Data Source API V2 in Spark SQL.

[[contract]] .ContinuousReader Contract [cols="1m,3",options="header",width="100%"] |=== | Method | Description

| commit a| [[commit]]

[source, java]¶

void commit(Offset end)¶

Commits the specified offset

Used exclusively when ContinuousExecution is requested to commit an epoch

| deserializeOffset a| [[deserializeOffset]]

[source, java]¶

Offset deserializeOffset(String json)¶

Deserializes an offset from JSON representation

Used when ContinuousExecution is requested to run a streaming query and commit an epoch

| getStartOffset a| [[getStartOffset]]

[source, java]¶

Offset getStartOffset()¶

NOTE: Used exclusively in tests.

| mergeOffsets a| [[mergeOffsets]]

[source, java]¶

Offset mergeOffsets(PartitionOffset[] offsets)¶

Used exclusively when ContinuousExecution is requested to addOffset

| needsReconfiguration a| [[needsReconfiguration]]

[source, java]¶

boolean needsReconfiguration()¶

Indicates that the reader needs reconfiguration (e.g. to generate new input partitions)

Used exclusively when ContinuousExecution is requested to run a streaming query in continuous mode

| setStartOffset a| [[setStartOffset]]

[source, java]¶

void setStartOffset(Optional start)¶

Used exclusively when ContinuousExecution is requested to run the streaming query in continuous mode.

|===

[[implementations]] .ContinuousReaders [cols="1,2",options="header",width="100%"] |=== | ContinuousReader | Description

| ContinuousMemoryStream | [[ContinuousMemoryStream]]

| RateStreamContinuousReader | [[RateStreamContinuousReader]]

| TextSocketContinuousReader | [[TextSocketContinuousReader]]

|===