Skip to content

Spark Logging

Apache Spark uses Apache Log4j 2 for logging.

conf/log4j2.properties

The default logging for Spark applications is in conf/log4j2.properties.

Use conf/log4j2.properties.template as a starting point.

Logging Levels

The valid logging levels are log4j's Levels (from most specific to least):

Name Description
OFF No events will be logged
FATAL A fatal event that will prevent the application from continuing
ERROR An error in the application, possibly recoverable
WARN An event that might possible lead to an error
INFO An event for informational purposes
DEBUG A general debugging event
TRACE A fine-grained debug message, typically capturing the flow through the application
ALL All events should be logged

The names of the logging levels are case-insensitive.

Turn Logging Off

The following sample conf/log4j2.properties turns all logging of Apache Spark (and Apache Hadoop) off.

# Set to debug or trace if log4j initialization fails
status = warn

# Name of the configuration
name = exploring-internals

# Console appender configuration
appender.console.type = Console
appender.console.name = consoleLogger
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{YYYY-MM-dd HH:mm:ss} [%t] %-5p %c:%L - %m%n
appender.console.target = SYSTEM_OUT

rootLogger.level = off
rootLogger.appenderRef.stdout.ref = consoleLogger

logger.spark.name = org.apache.spark
logger.spark.level = off

logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = off

Setting Default Log Level Programatically

Setting Default Log Level Programatically

Setting Log Levels in Spark Applications

In standalone Spark applications or while in Spark Shell session, use the following:

import org.apache.log4j.{Level, Logger}

Logger.getLogger(classOf[RackResolver]).getLevel
Logger.getLogger("org").setLevel(Level.OFF)