Skip to content

ContinuousQueuedDataReader

== [[ContinuousQueuedDataReader]] ContinuousQueuedDataReader

ContinuousQueuedDataReader is <> exclusively when ContinuousDataSourceRDD is requested to <>.

[[ContinuousRecord]] ContinuousQueuedDataReader uses two types of continuous records:

  • [[EpochMarker]] EpochMarker
  • [[ContinuousRow]] ContinuousRow (with the InternalRow at PartitionOffset)

=== [[next]] Fetching Next Row -- next Method

[source, scala]

next(): InternalRow

next...FIXME

NOTE: next is used when...FIXME

=== [[close]] Closing ContinuousQueuedDataReader -- close Method

[source, scala]

close(): Unit

NOTE: close is part of the https://docs.oracle.com/javase/8/docs/api/java/io/Closeable.html[java.io.Closeable] to close this stream and release any system resources associated with it.

close...FIXME

=== [[creating-instance]] Creating ContinuousQueuedDataReader Instance

ContinuousQueuedDataReader takes the following to be created:

  • [[partition]] ContinuousDataSourceRDDPartition
  • [[context]] TaskContext
  • [[dataQueueSize]] Size of the <>
  • [[epochPollIntervalMs]] epochPollIntervalMs

ContinuousQueuedDataReader initializes the <>.

=== [[internal-properties]] Internal Properties

[cols="30m,70",options="header",width="100%"] |=== | Name | Description

| coordinatorId a| [[coordinatorId]] Epoch Coordinator Identifier

Used when...FIXME

| currentOffset a| [[currentOffset]] PartitionOffset

Used when...FIXME

| dataReaderThread a| [[dataReaderThread]] <> daemon thread that is created and started immediately when ContinuousQueuedDataReader is <>

Used when...FIXME

| epochCoordEndpoint a| [[epochCoordEndpoint]] RpcEndpointRef of the <> per <>

Used when...FIXME

| epochMarkerExecutor a| [[epochMarkerExecutor]] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ScheduledExecutorService.html[java.util.concurrent.ScheduledExecutorService]

Used when...FIXME

| epochMarkerGenerator a| [[epochMarkerGenerator]] <>

Used when...FIXME

| reader a| [[reader]] InputPartitionReader

Used when...FIXME

| queue a| [[queue]] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ArrayBlockingQueue.html[java.util.concurrent.ArrayBlockingQueue] of <> (of the given <>)

Used when...FIXME

|===