Skip to content

The Internals of Spark Structured Streaming

Options

Initializing search

spark-structured-streaming-internals

Home
Internals
Features
High-Level APIs
Streaming Operators
Data Sources
Monitoring
Web UI
Demo

The Internals of Spark Structured Streaming

spark-structured-streaming-internals

Home
Internals
Internals
- Overview
  Overview
  - Overview
  - Internals of Streaming Queries
- Configuration Properties
- DataSource
- Execution Planning Strategies
  Execution Planning Strategies
- IncrementalExecution
- Logical Analysis Rules
  Logical Analysis Rules
  - ResolveWriteToStream
- Logical Operators
  Logical Operators
- Offsets and Metadata Checkpointing
  Offsets and Metadata Checkpointing
  - Offsets and Metadata Checkpointing
  - MetadataLog
  - HDFSMetadataLog
  - CommitLog
    CommitLog
    
    CommitLog
    
    CommitMetadata
  - OffsetSeqLog
    OffsetSeqLog
    
    OffsetSeqLog
    
    OffsetSeq
  - OffsetSeqMetadata
  - CheckpointFileManager
    CheckpointFileManager
    
    CheckpointFileManager
    
    FileContextBasedCheckpointFileManager
    
    FileSystemBasedCheckpointFileManager
    
    Offset
    
    StreamProgress
- Physical Operators
  Physical Operators
- ProgressReporter
- SQLConf
- StreamExecution
- StreamingDataWriterFactory
- StreamingQueryListenerBus
- StreamMetadata
- TriggerExecutors
  TriggerExecutors
  - ProcessingTimeExecutor
  - TriggerExecutor
- UnsupportedOperationChecker
- Misc
  Misc
Features
Features
- Data Sources
- Micro-Batch Execution
  Micro-Batch Execution
- RocksDB State Store
  RocksDB State Store
- Arbitrary Stateful Streaming Aggregation
  Arbitrary Stateful Streaming Aggregation
  - FlatMapGroupsWithStateExecHelper
  - GroupState
    GroupState
    
    GroupState
    
    GroupStateImpl
  - GroupStateTimeout
  - InputProcessor
  - StateManager
    StateManager
    
    StateManager
    
    StateManagerImplBase
    
    StateManagerImplV1
    
    StateManagerImplV2
- Batch Processing Time
- Streaming Aggregation
  Streaming Aggregation
- Streaming Join
  Streaming Join
- Streaming Watermark
  Streaming Watermark
- Streaming Deduplication
  Streaming Deduplication
- Streaming Limit
  Streaming Limit
- Stateful Stream Processing
  Stateful Stream Processing
  - BaseStateStoreRDD
  - ReadStateStoreRDD
  - StatefulOperatorPartitioning
  - StatefulOperatorStateInfo
  - StateSchemaCompatibilityChecker
  - StateStore
    StateStore
    
    HDFSBackedStateStore
    
    ReadStateStore
    
    StateStore
    
    StateStoreId
  - StateStoreConf
  - StateStoreCoordinator
    StateStoreCoordinator
    
    StateStoreCoordinator
    
    StateStoreCoordinatorRef
  - StateStoreCustomMetric
  - StateStoreMetrics
  - StateStoreOps
  - StateStoreProvider
    StateStoreProvider
    
    HDFSBackedStateStoreProvider
    
    StateStoreProvider
    
    StateStoreProviderId
  - StateStoreRDD
  - StreamingSessionWindowStateManager
- Continuous Execution
  Continuous Execution
  - ContinuousExecution
  - ContinuousReadSupport
  - ContinuousReader
  - EpochCoordinator
    EpochCoordinator
    
    EpochCoordinator RPC Endpoint
    
    EpochCoordinatorRef
    
    EpochTracker
  - ContinuousQueuedDataReader
    ContinuousQueuedDataReader
    
    ContinuousQueuedDataReader
    
    DataReaderThread
    
    EpochMarkerGenerator
  - PartitionOffset
  - ContinuousWriteRDD
  - ContinuousDataSourceRDD
- Testing Streaming Queries
- Web UI
High-Level APIs
High-Level APIs
Streaming Operators
Streaming Operators
Data Sources
Data Sources
- Console
  Console
  - ConsoleSinkProvider
  - ConsoleWriter
- File
  File
- Foreach
  Foreach
- ForeachBatch
  ForeachBatch
  - ForeachBatchSink
- Kafka
  Kafka
  - CachedKafkaConsumer
  - Consumer Strategies
    Consumer Strategies
    
    ConsumerStrategy
    
    SubscribeStrategy
  - KafkaBatch
  - KafkaBatchInputPartition
  - KafkaContinuousInputPartition
  - KafkaContinuousStream
  - KafkaDataConsumer
  - KafkaMicroBatchInputPartition
  - KafkaMicroBatchInputPartitionReader
  - KafkaMicroBatchStream
  - KafkaOffsetRangeCalculator
  - KafkaOffsetRangeLimit
  - KafkaOffsetReaders
    KafkaOffsetReaders
    
    KafkaOffsetReader
    
    KafkaOffsetReaderAdmin
    
    KafkaOffsetReaderConsumer
  - KafkaRelation
  - KafkaScan
  - KafkaSink
  - KafkaSource
  - KafkaSourceInitialOffsetWriter
  - KafkaSourceOffset
  - KafkaSourceProvider
  - KafkaSourceRDD
  - KafkaStreamingWrite
  - KafkaStreamWriterFactory
  - KafkaTable
  - Options
- Memory
  Memory
- Rate
  Rate
  - RateStreamProvider
  - RateStreamContinuousReader
- Rate Per Micro-Batch
  Rate Per Micro-Batch
  - Options Options
    Table of contents
    
    advanceMillisPerBatch
    
    numPartitions
    
    rowsPerBatch
    
    startTimestamp
  - RatePerMicroBatchProvider
  - RatePerMicroBatchStream
  - RatePerMicroBatchTable
- Text Socket
  Text Socket
  - TextSocketSourceProvider
  - TextSocketSource
Monitoring
Monitoring
- StreamingQueryListener
- StreamingQueryProgress
  StreamingQueryProgress
- StreamingQueryStatus
- MetricsReporter
- Logging
Web UI
Web UI
Demo
Demo

Table of contents

advanceMillisPerBatch
numPartitions
rowsPerBatch
startTimestamp

Options¶

advanceMillisPerBatch¶

default: 1000

numPartitions¶

default: SparkSession.active.sparkContext.defaultParallelism

rowsPerBatch¶

default: 0

startTimestamp¶

default: 0

Previous Rate Per Micro-Batch Data Source

Next RatePerMicroBatchProvider

Copyright © 2022 Jacek Laskowski

Made with Material for MkDocs Insiders