Skip to content

Spark Properties and spark-defaults.conf Properties File

Spark properties are the means of tuning the execution environment of a Spark application.

The default Spark properties file is <$SPARK_HOME/conf/spark-defaults.conf>> that could be overriden using spark-submit with the[--properties-file] command-line option.

.Environment Variables [options="header",width="100%"] |=== | Environment Variable | Default Value | Description | SPARK_CONF_DIR | $\{SPARK_HOME}/conf | Spark's configuration directory (with spark-defaults.conf) |===

TIP: Read the official documentation of Apache Spark on[Spark Configuration].

=== [[spark-defaults-conf]] spark-defaults.conf -- Default Spark Properties File

spark-defaults.conf (under SPARK_CONF_DIR or $SPARK_HOME/conf) is the default properties file with the Spark properties of your Spark applications.

NOTE: spark-defaults.conf is loaded by[AbstractCommandBuilder's loadPropertiesFile internal method].

=== [[getDefaultPropertiesFile]] Calculating Path of Default Spark Properties -- Utils.getDefaultPropertiesFile method

[source, scala]

getDefaultPropertiesFile(env: Map[String, String] = sys.env): String

getDefaultPropertiesFile calculates the absolute path to spark-defaults.conf properties file that can be either in directory specified by SPARK_CONF_DIR environment variable or $SPARK_HOME/conf directory.

NOTE: getDefaultPropertiesFile is part of private[spark] org.apache.spark.util.Utils object.

Last update: 2020-11-21