Skip to content

Configuration Properties

spark.kafka.consumer.cache.capacity

The maximum number of consumers cached. Please note it's a soft limit (check Structured Streaming Kafka integration guide for further details).

Default: 64

Used in InternalKafkaConsumerPool

spark.kafka.consumer.cache.evictorThreadRunInterval

The interval of time (in millis) between runs of the idle evictor thread for consumer pool. When non-positive, no idle evictor thread will be run

Default: 1m

spark.kafka.consumer.cache.jmx.enable

Enable or disable JMX for pools created with this configuration instance.

Default: false

spark.kafka.consumer.cache.timeout

The minimum amount of time (in millis) a consumer may sit idle in the pool before it is eligible for eviction by the evictor. When non-positive, no consumers will be evicted from the pool due to idle time alone.

Default: 5m

spark.kafka.consumer.cache.evictorThreadRunInterval

The interval of time (in millis) between runs of the idle evictor thread for consumer pool. When non-positive, no idle evictor thread will be run.

Default: 1m

spark.kafka.consumer.fetchedData.cache.timeout

The minimum amount of time (in millis) a fetched data may sit idle in the pool before it is eligible for eviction by the evictor. When non-positive, no fetched data will be evicted from the pool due to idle time alone.

Default: 5m

spark.kafka.producer.cache.evictorThreadRunInterval

The interval of time (in millis) between runs of the idle evictor thread for producer pool. When non-positive, no idle evictor thread will be run.

Default: 1m

spark.kafka.producer.cache.timeout

The expire time (in millis) to remove the unused producers.

Default: 10m

Used when: