LogCleaner¶

LogCleaner is created and started immediately for LogManager when started with log.cleaner.enable configuration property enabled.

Creating Instance¶

LogCleaner takes the following to be created:

• CleanerConfig
• Log directories
• UnifiedLogs by TopicPartition (Pool[TopicPartition, UnifiedLog])
• LogDirFailureChannel
• Time

LogCleaner is created when:

• LogManager is requested to startupWithConfigOverrides (with log.cleaner.enable enabled)

LogCleanerManager¶

LogCleaner creates a LogCleanerManager (with the log directories and UnifiedLogs) when created.

KafkaMetricsGroup¶

LogCleaner is a KafkaMetricsGroup.

BrokerReconfigurable¶

LogCleaner is a BrokerReconfigurable.

Starting Up¶

startup(): Unit

startup prints out the following INFO message to the logs:

Starting the log cleaner

startup creates log.cleaner.threads of CleanerThreads that are started immediately.

startup adds the CleanerThreads to the cleaners registry.

startup is used when:

• LogCleaner is requested to reconfigure
• LogManager is requested to startupWithConfigOverrides (with log.cleaner.enable enabled)

Creating CleanerConfig¶

cleanerConfig(
  config: KafkaConfig): CleanerConfig

cleanerConfig creates a CleanerConfig with the configuration properties from the given KafkaConfig.

cleanerConfig is used when:

• LogCleaner is requested to validateReconfiguration and reconfigure
• LogManager utility is used to create a LogManager

CleanerThreads¶

LogCleaner uses CleanerThreads for log cleaning.

LogCleaner creates log.cleaner.threads of CleanerThreads when started up. They are shut down when shutdown.

Use the following metrics to monitor the threads:

• cleaner-recopy-percent
• DeadThreadCount

Logging¶

Enable ALL logging level for kafka.log.LogCleaner logger to see what happens inside.

Add the following line to config/log4j.properties:

log4j.logger.kafka.log.LogCleaner=ALL

Refer to Logging.

log4j.appender.cleanerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.cleanerAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.cleanerAppender.File=${kafka.logs.dir}/log-cleaner.log
log4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender
log4j.additivity.kafka.log.LogCleaner=false