SparkSubmitOptionParser — spark-submit’s Command-Line Parser

SparkSubmitOptionParser is the parser of spark-submit's command-line options.

Table 1. spark-submit Command-Line Options
Command-Line Option Description



The main class to run (as mainClass internal attribute).

--conf [prop=value] or -c [prop=value]

All =-separated values end up in conf potentially overriding existing settings. Order on command-line matters.


deployMode internal property


spark.driver.extraClassPath in conf — the driver class path



spark.driver.extraJavaOptions in conf — the driver VM options


spark.driver.extraLibraryPath in conf — the driver native library path


spark.driver.memory in conf





--help or -h

The option is added to sparkArgs




The option and a value are added to sparkArgs


master internal property





--properties-file [FILE]

propertiesFile internal property. Refer to Custom Spark Properties File — --properties-file command-line option.






The option and a value are added to sparkArgs




The option is added to sparkArgs

--verbose or -v


The option is added to sparkArgs

SparkSubmitOptionParser Callbacks

SparkSubmitOptionParser is supposed to be overriden for the following capabilities (as callbacks).

Table 2. Callbacks
Callback Description


Executed when an option with an argument is parsed.


Executed when an unrecognized option is parsed.


Executed for the command-line arguments that handle and handleUnknown callbacks have not processed.

SparkSubmitOptionParser belongs to org.apache.spark.launcher Scala package and spark-launcher Maven/sbt module.

org.apache.spark.launcher.SparkSubmitArgumentsParser is a custom SparkSubmitOptionParser.

Parsing Command-Line Arguments — parse Method

final void parse(List<String> args)

parse parses a list of command-line arguments.

parse calls handle callback whenever it finds a known command-line option or a switch (a command-line option with no parameter). It calls handleUnknown callback for unrecognized command-line options.

parse keeps processing command-line arguments until handle or handleUnknown callback return false or all command-line arguments have been consumed.

Ultimately, parse calls handleExtraArgs callback.