Skip to content

Deployment Environments

Spark Deployment Environments (Run Modes):

  • local/[local]
  •[clustered] **[Spark Standalone] ** Spark on Apache Mesos ** yarn/[Spark on Hadoop YARN]

A Spark application is composed of the driver and executors that can run locally (on a single JVM) or using cluster resources (like CPU, RAM and disk that are managed by a cluster manager).

NOTE: You can specify where to run the driver using the[deploy mode] (using --deploy-mode option of spark-submit or spark.submit.deployMode Spark property).

== [[master-urls]] Master URLs

Spark supports the following master URLs (see[private object SparkMasterRegex]):

  • local, local[N] and local[{asterisk}] for local/[Spark local]
  • local[N, maxRetries] for local/[Spark local-with-retries]
  • local-cluster[N, cores, memory] for simulating a Spark cluster of N executors (threads), cores CPUs and memory locally (aka Spark local-cluster)
  • spark://host:port,host1:port1,... for connecting to[Spark Standalone cluster(s)]
  • mesos:// for spark-mesos/[Spark on Mesos cluster]
  • yarn for yarn/[Spark on YARN]

You can specify the master URL of a Spark application as follows:

  1.[spark-submit's --master command-line option],

  2.[spark.master Spark property],

  3. When creating a[SparkContext (using setMaster method)],

  4. When creating a[SparkSession (using master method of the builder interface)].

Last update: 2020-11-27
Back to top