Skip to content

MemoryStreamBase -- Base Contract for Memory Sources

MemoryStreamBase is the <> of...FIXME

[[contract]] .MemoryStreamBase Contract [cols="30m,70",options="header",width="100%"] |=== | Method | Description

| addData a| [[addData]]

[source, scala]

addData( data: TraversableOnce[A]): Offset

| logicalPlan a| [[logicalPlan]]

[source, scala]

logicalPlan: LogicalPlan

|===

[[implementations]] .MemoryStreamBases [cols="30,70",options="header",width="100%"] |=== | MemoryStreamBase | Description

| ContinuousMemoryStream | [[ContinuousMemoryStream]]

| MemoryStream | [[MemoryStream]] MicroBatchReader for Micro-Batch Stream Processing

|===

=== [[creating-instance]] Creating MemoryStreamBase Instance

MemoryStreamBase takes the following to be created:

  • [[sqlContext]] SQLContext

NOTE: MemoryStreamBase is a Scala abstract class and cannot be <> directly. It is created indirectly for the <>.

=== [[toDS]] Creating Streaming Dataset -- toDS Method

[source, scala]

toDS(): Dataset[A]

toDS simply creates a Dataset (for the <> and the <>)

=== [[toDF]] Creating Streaming DataFrame -- toDF Method

[source, scala]

toDF(): DataFrame

toDF simply creates a Dataset of rows (for the <> and the <>)

=== [[internal-properties]] Internal Properties

[cols="30m,70",options="header",width="100%"] |=== | Name | Description

| attributes a| [[attributes]] Schema attributes of the <> (Seq[AttributeReference])

Used when...FIXME

| encoder a| [[encoder]] Spark SQL's ExpressionEncoder for the data

Used when...FIXME

|===