Skip to content
The Internals of Spark Structured Streaming
Streaming Deduplication
Initializing search
spark-structured-streaming-internals
Home
Internals
Features
High-Level APIs
Streaming Operators
Data Sources
Monitoring
Web UI
Demo
The Internals of Spark Structured Streaming
spark-structured-streaming-internals
Home
Internals
Internals
Overview
Overview
Overview
Internals of Streaming Queries
Configuration Properties
DataSource
Execution Planning Strategies
Execution Planning Strategies
FlatMapGroupsWithStateStrategy
StatefulAggregationStrategy
StreamingDeduplicationStrategy
StreamingGlobalLimitStrategy
StreamingJoinStrategy
StreamingRelationStrategy
IncrementalExecution
Logical Analysis Rules
Logical Analysis Rules
ResolveWriteToStream
Logical Operators
Logical Operators
ContinuousExecutionRelation
Deduplicate
EventTimeWatermark
FlatMapGroupsWithState
StreamingDataSourceV2Relation
StreamingExecutionRelation
StreamingRelation
StreamingRelationV2
WriteToContinuousDataSource
WriteToDataSourceV2
WriteToMicroBatchDataSource
WriteToStream
WriteToStreamStatement
Offsets and Metadata Checkpointing
Offsets and Metadata Checkpointing
Offsets and Metadata Checkpointing
MetadataLog
HDFSMetadataLog
CommitLog
CommitLog
CommitLog
CommitMetadata
OffsetSeqLog
OffsetSeqLog
OffsetSeqLog
OffsetSeq
OffsetSeqMetadata
CheckpointFileManager
CheckpointFileManager
CheckpointFileManager
FileContextBasedCheckpointFileManager
FileSystemBasedCheckpointFileManager
Offset
StreamProgress
Physical Operators
Physical Operators
EventTimeWatermarkExec
FlatMapGroupsWithStateExec
MicroBatchScanExec
SessionWindowStateStoreRestoreExec
SessionWindowStateStoreSaveExec
StateStoreReader
StateStoreRestoreExec
StateStoreSaveExec
StateStoreWriter
StatefulOpClusteredDistribution
StatefulOperator
StreamingDeduplicateExec
StreamingGlobalLimitExec
StreamingRelationExec
StreamingSymmetricHashJoinExec
WatermarkSupport
WriteToContinuousDataSourceExec
WriteToDataSourceV2Exec
ProgressReporter
SQLConf
StreamExecution
StreamingDataWriterFactory
StreamingQueryListenerBus
StreamMetadata
TriggerExecutors
TriggerExecutors
ProcessingTimeExecutor
TriggerExecutor
UnsupportedOperationChecker
Misc
Misc
AcceptsLatestSeenOffsetHandler
AvailableNowDataStreamWrapper
StreamingQueryWrapper
Features
Features
Arbitrary State API v2
Data Sources
Micro-Batch Execution
Micro-Batch Execution
MicroBatchExecution
MicroBatchWrite
MicroBatchWriterFactory
RocksDB State Store
RocksDB State Store
RocksDB
RocksDBConf
RocksDBFileManager
RocksDBMetrics
RocksDBStateStore
RocksDBStateStoreProvider
Arbitrary Stateful Streaming Aggregation
Arbitrary Stateful Streaming Aggregation
FlatMapGroupsWithStateExecHelper
GroupState
GroupState
GroupState
GroupStateImpl
GroupStateTimeout
InputProcessor
StateManager
StateManager
StateManager
StateManagerImplBase
StateManagerImplV1
StateManagerImplV2
Batch Processing Time
Streaming Aggregation
Streaming Aggregation
StreamingAggregationStateManager
StreamingAggregationStateManagerBaseImpl
StreamingAggregationStateManagerImplV2
Streaming Join
Streaming Join
JoinStateWatermarkPredicate
JoinStateWatermarkPredicates
KeyToNumValuesStore
KeyWithIndexToValueStore
OneSideHashJoiner
StateStoreAwareZipPartitionsHelper
StateStoreAwareZipPartitionsRDD
StateStoreHandler
StreamingJoinHelper
StreamingSymmetricHashJoinHelper
SymmetricHashJoinStateManager
Streaming Watermark
Streaming Watermark
EventTimeStatsAccum
EventTimeStats
WatermarkTracker
Streaming Deduplication
Streaming Deduplication
Streaming Limit
Streaming Limit
Stateful Stream Processing
Stateful Stream Processing
BaseStateStoreRDD
ReadStateStoreRDD
StatefulOperatorPartitioning
StatefulOperatorStateInfo
StateSchemaCompatibilityChecker
StateStore
StateStore
HDFSBackedStateStore
ReadStateStore
StateStore
StateStoreId
StateStoreConf
StateStoreCoordinator
StateStoreCoordinator
StateStoreCoordinator
StateStoreCoordinatorRef
StateStoreCustomMetric
StateStoreMetrics
StateStoreOps
StateStoreProvider
StateStoreProvider
HDFSBackedStateStoreProvider
StateStoreProvider
StateStoreProviderId
StateStoreRDD
StreamingSessionWindowStateManager
Continuous Execution
Continuous Execution
ContinuousExecution
ContinuousReadSupport
ContinuousReader
EpochCoordinator
EpochCoordinator
EpochCoordinator RPC Endpoint
EpochCoordinatorRef
EpochTracker
ContinuousQueuedDataReader
ContinuousQueuedDataReader
ContinuousQueuedDataReader
DataReaderThread
EpochMarkerGenerator
PartitionOffset
ContinuousWriteRDD
ContinuousDataSourceRDD
Testing Streaming Queries
Web UI
High-Level APIs
High-Level APIs
AcceptsLatestSeenOffset
ContinuousStream
DataStreamReader
DataStreamWriter
MicroBatchStream
OutputMode
ReadLimit
ReportsSourceMetrics
Sink
Source
SparkDataStream
StreamingQuery
StreamingQueryManager
StreamingWrite
StreamSinkProvider
StreamSourceProvider
SupportsAdmissionControl
SupportsTriggerAvailableNow
Trigger
Streaming Operators
Streaming Operators
crossJoin
dropDuplicates
explain
flatMapGroupsWithState
groupBy
groupByKey
join
joinWith
mapGroupsWithState
window
withWatermark
writeStream
Data Sources
Data Sources
Console
Console
ConsoleSinkProvider
ConsoleWriter
File
File
FileStreamSource
FileStreamSink
CompactibleFileStreamLog
SinkFileStatus
ManifestFileCommitProtocol
MetadataLogFileIndex
FileStreamSourceCleaner
FileStreamSourceLog
FileStreamSinkLog
Foreach
Foreach
ForeachSink
ForeachWriter
ForeachWriterProvider
ForeachWriterTable
ForeachBatch
ForeachBatch
ForeachBatchSink
Kafka
Kafka
CachedKafkaConsumer
Consumer Strategies
Consumer Strategies
ConsumerStrategy
SubscribeStrategy
KafkaBatch
KafkaBatchInputPartition
KafkaContinuousInputPartition
KafkaContinuousStream
KafkaDataConsumer
KafkaMicroBatchInputPartition
KafkaMicroBatchInputPartitionReader
KafkaMicroBatchStream
KafkaOffsetRangeCalculator
KafkaOffsetRangeLimit
KafkaOffsetReaders
KafkaOffsetReaders
KafkaOffsetReader
KafkaOffsetReaderAdmin
KafkaOffsetReaderConsumer
KafkaRelation
KafkaScan
KafkaSink
KafkaSource
KafkaSourceInitialOffsetWriter
KafkaSourceOffset
KafkaSourceProvider
KafkaSourceRDD
KafkaStreamingWrite
KafkaStreamWriterFactory
KafkaTable
Options
Memory
Memory
ContinuousMemoryStream
MemoryPlan
MemorySink
MemoryStream
MemoryStreamBase
MemoryStreamingWrite
Rate
Rate
RateStreamProvider
RateStreamContinuousReader
Rate Per Micro-Batch
Rate Per Micro-Batch
Options
RatePerMicroBatchProvider
RatePerMicroBatchStream
RatePerMicroBatchTable
State
State
Text Socket
Text Socket
TextSocketSourceProvider
TextSocketSource
Monitoring
Monitoring
StreamingQueryListener
StreamingQueryProgress
StreamingQueryProgress
StreamingQueryProgress
StateOperatorProgress
ExecutionStats
SourceProgress
SinkProgress
StreamingQueryStatus
MetricsReporter
Logging
Web UI
Web UI
StreamingQueryPage
StreamingQueryStatisticsPage
StreamingQueryStatusListener
StreamingQueryStatusStore
StreamingQueryTab
Demo
Demo
Arbitrary Stateful Streaming Aggregation with KeyValueGroupedDataset.flatMapGroupsWithState Operator
current_timestamp Function For Processing Time in Streaming Queries
Custom Streaming Sink (and Monitoring SQL Queries in web UI)
Deep Dive into FileStreamSink
Exploring Checkpointed State
groupByKey Streaming Aggregation in Update Mode
Internals of FlatMapGroupsWithStateExec Physical Operator
Kafka Data Source
Streaming Windowed Aggregation
RocksDB State Store for Streaming Aggregation
Stream-Stream Inner Join
Streaming Query for Running Counts (Socket Source and Complete Output Mode)
Streaming Watermark
Streaming Watermark with Aggregation in Append Output Mode
StateStoreSaveExec with Complete Output Mode
StateStoreSaveExec with Update Output Mode
Using StreamingQueryManager for Query Termination Management
Using File Streaming Source
Home
Features
Streaming Deduplication
Streaming Deduplication
¶
Streaming Deduplication
is...FIXME
Back to top