Skip to content

The Internals of Spark Structured Streaming

Logical Operators for Streaming Queries

Initializing search

Github

Home
Internals
Features
High-Level APIs
Streaming Operators
Data Sources
Monitoring
Web UI
Demo

The Internals of Spark Structured Streaming

Github

Home
Internals
Internals
- Overview
- Internals of Streaming Queries
- Configuration Properties
- DataSource
- Execution Planning Strategies
  Execution Planning Strategies
- IncrementalExecution
- Logical Analysis Rules
  Logical Analysis Rules
  - ResolveWriteToStream
- Logical Operators
  Logical Operators
- Offsets and Metadata Checkpointing
  Offsets and Metadata Checkpointing
  - Offsets and Metadata Checkpointing
  - MetadataLog
  - HDFSMetadataLog
  - CommitLog
    CommitLog
    
    CommitLog
    
    CommitMetadata
  - OffsetSeqLog
    OffsetSeqLog
    
    OffsetSeqLog
    
    OffsetSeq
  - OffsetSeqMetadata
  - CheckpointFileManager
    CheckpointFileManager
    
    CheckpointFileManager
    
    FileContextBasedCheckpointFileManager
    
    FileSystemBasedCheckpointFileManager
    
    Offset
    
    StreamProgress
- Physical operators
  Physical operators
- ProgressReporter
- SQLConf
- StreamExecution
- StreamingDataWriterFactory
- StreamingQueryListenerBus
- StreamMetadata
- TriggerExecutors
  TriggerExecutors
  - ProcessingTimeExecutor
  - TriggerExecutor
- UnsupportedOperationChecker
- Misc
  Misc
Features
Features
- Real-Time Mode
  Real-Time Mode
- Async Progress Tracking
  Async Progress Tracking
  - AsyncProgressTrackingMicroBatchExecution
- Arbitrary State API v2
- Data Sources
- Micro-Batch Execution
  Micro-Batch Execution
- RocksDB State Store
  RocksDB State Store
- Arbitrary Stateful Streaming Aggregation
  Arbitrary Stateful Streaming Aggregation
  - FlatMapGroupsWithStateExecHelper
  - GroupState
    GroupState
    
    GroupState
    
    GroupStateImpl
  - GroupStateTimeout
  - InputProcessor
  - StateManager
    StateManager
    
    StateManager
    
    StateManagerImplBase
    
    StateManagerImplV1
    
    StateManagerImplV2
- Batch Processing Time
- Streaming Aggregation
  Streaming Aggregation
- Streaming Deduplication
  Streaming Deduplication
- Streaming Join
  Streaming Join
- Streaming Limit
  Streaming Limit
- Streaming Watermark
  Streaming Watermark
- Stateful Stream Processing
  Stateful Stream Processing
  - BaseStateStoreRDD
  - ReadStateStoreRDD
  - StatefulOperatorPartitioning
  - StatefulOperatorStateInfo
  - StateSchemaCompatibilityChecker
  - StateStore
    StateStore
    
    HDFSBackedStateStore
    
    ReadStateStore
    
    StateStore
    
    StateStoreId
  - StateStoreConf
  - StateStoreCoordinator
    StateStoreCoordinator
    
    StateStoreCoordinator
    
    StateStoreCoordinatorRef
  - StateStoreCustomMetric
  - StateStoreMetrics
  - StateStoreOps
  - StateStoreProvider
    StateStoreProvider
    
    HDFSBackedStateStoreProvider
    
    StateStoreProvider
    
    StateStoreProviderId
  - StateStoreRDD
  - StreamingSessionWindowStateManager
- Continuous Execution
  Continuous Execution
  - ContinuousExecution
  - ContinuousReadSupport
  - ContinuousReader
  - EpochCoordinator
    EpochCoordinator
    
    EpochCoordinator RPC Endpoint
    
    EpochCoordinatorRef
    
    EpochTracker
  - ContinuousQueuedDataReader
    ContinuousQueuedDataReader
    
    ContinuousQueuedDataReader
    
    DataReaderThread
    
    EpochMarkerGenerator
  - PartitionOffset
  - ContinuousWriteRDD
  - ContinuousDataSourceRDD
- Testing Streaming Queries
High-Level APIs
High-Level APIs
Streaming Operators
Streaming Operators
Data Sources
Data Sources
- Console
  Console
  - ConsoleSinkProvider
  - ConsoleWriter
- File
  File
- Foreach
  Foreach
- ForeachBatch
  ForeachBatch
  - ForeachBatchSink
- Kafka
  Kafka
  - Options
  - Consumer Strategies
    Consumer Strategies
    
    ConsumerStrategy
    
    SubscribeStrategy
  - KafkaOffsetReaders
    KafkaOffsetReaders
    
    KafkaOffsetReader
    
    KafkaOffsetReaderAdmin
    
    KafkaOffsetReaderConsumer
  - CachedKafkaConsumer
  - KafkaBatch
  - KafkaBatchInputPartition
  - KafkaContinuousInputPartition
  - KafkaContinuousStream
  - KafkaDataConsumer
  - KafkaMicroBatchInputPartition
  - KafkaMicroBatchInputPartitionReader
  - KafkaMicroBatchStream
  - KafkaOffsetRangeCalculator
  - KafkaOffsetRangeLimit
  - KafkaRelation
  - KafkaScan
  - KafkaSink
  - KafkaSource
  - KafkaSourceInitialOffsetWriter
  - KafkaSourceOffset
  - KafkaSourceProvider
  - KafkaSourceRDD
  - KafkaStreamingWrite
  - KafkaStreamWriterFactory
  - KafkaTable
- Memory
  Memory
- Rate
  Rate
  - RateStreamContinuousReader
  - RateStreamProvider
- Rate Per Micro-Batch
  Rate Per Micro-Batch
- State
  State
- Socket
  Socket
  - TextSocketSource
  - TextSocketSourceProvider
Monitoring
Monitoring
- StreamingQueryListener
- StreamingQueryProgress
  StreamingQueryProgress
- StreamingQueryStatus
- MetricsReporter
- Logging
Web UI
Web UI
Demo
Demo

Home
Internals
Logical Operators

Logical Operators for Streaming Queries¶

Made with Material for MkDocs Insiders