Skip to content

SQLConf

SQLConf is an internal configuration store for parameters and hints used to configure a Spark Structured Streaming application (and Spark SQL applications in general).

Tip

Find out more on SQLConf in The Internals of Spark SQL.

FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION

spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion

STATEFUL_OPERATOR_USE_STRICT_DISTRIBUTION

spark.sql.streaming.statefulOperator.useStrictDistribution

stateStoreProviderClass

spark.sql.streaming.stateStore.providerClass

STREAMING_AGGREGATION_STATE_FORMAT_VERSION

spark.sql.streaming.aggregation.stateFormatVersion

streamingFileCommitProtocolClass

spark.sql.streaming.commitProtocolClass configuration property

Used when FileStreamSink is requested to "add" a batch of data

streamingMetricsEnabled

spark.sql.streaming.metricsEnabled configuration property

streamingNoDataMicroBatchesEnabled

spark.sql.streaming.noDataMicroBatches.enabled

Used when:

streamingNoDataProgressEventInterval

spark.sql.streaming.noDataProgressEventInterval

Used when:

fileSinkLogCleanupDelay

spark.sql.streaming.fileSink.log.cleanupDelay configuration property

Used when FileStreamSinkLog is created

fileSinkLogDeletion

spark.sql.streaming.fileSink.log.deletion configuration property

Used when FileStreamSinkLog is created

fileSinkLogCompactInterval

spark.sql.streaming.fileSink.log.compactInterval configuration property

Used when FileStreamSinkLog is created

minBatchesToRetain

spark.sql.streaming.minBatchesToRetain

Used when:

useDeprecatedKafkaOffsetFetching

spark.sql.streaming.kafka.useDeprecatedOffsetFetching

Review Me

[[accessor-methods]] .SQLConf's Property Accessor Methods [cols="1,1",options="header",width="100%"] |=== | Method Name / Property | Description

| continuousStreamingExecutorQueueSize

spark.sql.streaming.continuous.executorQueueSize

a| [[continuousStreamingExecutorQueueSize]] Used when:

  • DataSourceV2ScanExec leaf physical operator is requested for the input RDDs (and creates a <>)

  • ContinuousCoalesceExec unary physical operator is requested to execute

| continuousStreamingExecutorPollIntervalMs

spark.sql.streaming.continuous.executorPollIntervalMs

a| [[continuousStreamingExecutorPollIntervalMs]] Used exclusively when DataSourceV2ScanExec leaf physical operator is requested for the input RDDs (and creates a <>)

| disabledV2StreamingMicroBatchReaders

spark.sql.streaming.disabledV2MicroBatchReaders

a| [[disabledV2StreamingMicroBatchReaders]] Used exclusively when MicroBatchExecution is requested for the <> (of a streaming query)

| fileSourceLogDeletion

spark.sql.streaming.fileSource.log.deletion

a| [[fileSourceLogDeletion]][[FILE_SOURCE_LOG_DELETION]] Used exclusively when FileStreamSourceLog is requested for the isDeletingExpiredLog

| fileSourceLogCleanupDelay

spark.sql.streaming.fileSource.log.cleanupDelay

a| [[fileSourceLogCleanupDelay]][[FILE_SOURCE_LOG_CLEANUP_DELAY]] Used exclusively when FileStreamSourceLog is requested for the fileCleanupDelayMs

| fileSourceLogCompactInterval

spark.sql.streaming.fileSource.log.compactInterval

a| [[fileSourceLogCompactInterval]][[FILE_SOURCE_LOG_COMPACT_INTERVAL]] Used exclusively when FileStreamSourceLog is requested for the default compaction interval

| stateStoreMinDeltasForSnapshot

spark.sql.streaming.stateStore.minDeltasForSnapshot

a| [[stateStoreMinDeltasForSnapshot]] Used (as StateStoreConf.minDeltasForSnapshot) exclusively when HDFSBackedStateStoreProvider is requested to doSnapshot

| STREAMING_CHECKPOINT_FILE_MANAGER_CLASS

spark.sql.streaming.checkpointFileManagerClass a| [[STREAMING_CHECKPOINT_FILE_MANAGER_CLASS]] Used exclusively when CheckpointFileManager helper object is requested to create a CheckpointFileManager

| STREAMING_MULTIPLE_WATERMARK_POLICY

spark.sql.streaming.multipleWatermarkPolicy

a| [[STREAMING_MULTIPLE_WATERMARK_POLICY]]

| streamingPollingDelay

spark.sql.streaming.pollingDelay

a| [[streamingPollingDelay]][[STREAMING_POLLING_DELAY]] Used exclusively when StreamExecution is created

| streamingProgressRetention

spark.sql.streaming.numRecentProgressUpdates

a| [[streamingProgressRetention]][[STREAMING_PROGRESS_RETENTION]] Used exclusively when ProgressReporter is requested to update progress of streaming query (and possibly remove an excess)

|===