SparkSubmitArguments¶

SparkSubmitArguments is created  for SparkSubmit to parseArguments.

SparkSubmitArguments is a custom SparkSubmitArgumentsParser to handle the command-line arguments of spark-submit script that the actions use for execution (possibly with the explicit env environment).

SparkSubmitArguments is created when launching spark-submit script with only args passed in and later used for printing the arguments in verbose mode.

Creating Instance¶

SparkSubmitArguments takes the following to be created:

• Arguments (Seq[String])
• Environment Variables (default: sys.env)

SparkSubmitArguments is created when:

• SparkSubmit is requested to parseArguments

Action¶

action: SparkSubmitAction

action is used by SparkSubmit to determine what to do when executed.

action can be one of the following SparkSubmitActions:

action is undefined (null) by default (when SparkSubmitAction is created).

action is validated when validateArguments.

Command-Line Options¶

--files¶

• Configuration Property: spark.files
• Configuration Property (Spark on YARN): spark.yarn.dist.files

Printed out to standard output for --verbose option

When SparkSubmit is requested to prepareSubmitEnvironment, the files are:

• resolveGlobPaths
• downloadFileList
• renameResourcesToLocalFS
• downloadResource

Loading Spark Properties¶

loadEnvironmentArguments(): Unit

loadEnvironmentArguments loads the Spark properties for the current execution of spark-submit.

loadEnvironmentArguments reads command-line options first followed by Spark properties and System's environment variables.

Option Handling¶

handle(
  opt: String,
  value: String): Boolean

handle parses the input opt argument and assigns the given value to corresponding properties.

In the end, handle returns whether it was executed for any action but PRINT_VERSION.

maybeRemote¶

maybeRemote: Option[String]

maybeRemote is the value of the following (in this order of precedence):

1. --remote command-line option
2. spark.remote configuration property
3. SPARK_REMOTE environment variable

maybeRemote must not be used alongside master or deploy mode. They are exclusive.

If maybeRemote is not specified, maybeMaster is used.

maybeRemote can be displayed when SparkSubmitArguments is requested to toString.

mergeDefaultSparkProperties¶

mergeDefaultSparkProperties(): Unit

mergeDefaultSparkProperties merges Spark properties from the default Spark properties file, i.e. spark-defaults.conf with those specified through --conf command-line option.

isPython¶

isPython: Boolean = false

isPython indicates whether the application resource is a PySpark application (a Python script or pyspark shell).

isPython is isPython when SparkSubmitArguments is requested to handle a unknown option.

Client Deploy Mode¶

With isPython flag enabled, SparkSubmit determines the mainClass (and the childArgs) based on the primaryResource.