|Refer to Spark Configuration in the official documentation for an extensive coverage of how to configure Spark and user programs.|
There are three ways to configure Spark and user programs:
Spark Properties - use Web UI to learn the current properties.
Every user program starts with creating an instance of
SparkConf that holds the master URL to connect to (
spark.master), the name for your Spark application (that is later displayed in web UI and becomes
spark.app.name) and other Spark properties required for proper runs. The instance of
SparkConf can be used to create SparkContext.
Start Spark shell with
You can query for the values of Spark properties in Spark shell as follows:
scala> sc.getConf.getOption("spark.local.dir") res0: Option[String] = None scala> sc.getConf.getOption("spark.app.name") res1: Option[String] = Some(Spark shell) scala> sc.getConf.get("spark.master") res2: String = local[*]
There are the following places where a Spark application looks for Spark properties (in the order of importance from the least important to the most important):
The default Spark configuration is created when you execute the following code:
import org.apache.spark.SparkConf val conf = new SparkConf
It simply loads
spark.* system properties.
You can use
conf.getAll to have the
spark.* system properties loaded printed out.
scala> conf.getAll res0: Array[(String, String)] = Array((spark.app.name,Spark shell), (spark.jars,""), (spark.master,local[*]), (spark.submit.deployMode,client)) scala> conf.toDebugString res1: String = spark.app.name=Spark shell spark.jars= spark.master=local[*] spark.submit.deployMode=client scala> println(conf.toDebugString) spark.app.name=Spark shell spark.jars= spark.master=local[*] spark.submit.deployMode=client
getAppId gives spark.app.id Spark property or reports
NoSuchElementException if not set.
|Spark Property||Default Value||Description|
Unique identifier of a Spark application that Spark uses to uniquely identify metric sources.