SparkSubmitCommandBuilder¶

SparkSubmitCommandBuilder is an AbstractCommandBuilder.

SparkSubmitCommandBuilder is used to build a command that spark-submit and SparkLauncher use to launch a Spark application.

SparkSubmitCommandBuilder uses the first argument to distinguish the shells:

1. pyspark-shell-main
2. sparkr-shell-main
3. run-example

SparkSubmitCommandBuilder parses command-line arguments using OptionParser (which is a spark-submit-SparkSubmitOptionParser.md[SparkSubmitOptionParser]). OptionParser comes with the following methods:

1. handle to handle the known options (see the table below). It sets up master, deployMode, propertiesFile, conf, mainClass, sparkArgs internal properties.

2. handleUnknown to handle unrecognized options that usually lead to Unrecognized option error message.

3. handleExtraArgs to handle extra arguments that are considered a Spark application's arguments.

pyspark-shell-main Application Resource¶

When bin/pyspark shell script (and bin\pyspark2.cmd) are launched, they use bin/spark-submit with pyspark-shell-main application resource as the first argument (followed by --name "PySparkShell" option among the others).

pyspark-shell-main is used when:

• SparkSubmitCommandBuilder is created and then requested to build a command (buildPySparkShellCommand actually)

Building Command¶

List<String> buildCommand(
  Map<String, String> env)

buildCommand branches off based on the application resource.

buildPySparkShellCommand¶

List<String> buildPySparkShellCommand(
  Map<String, String> env)

buildPySparkShellCommand sets the application resource as pyspark-shell.

buildPySparkShellCommand constructEnvVarArgs with the given env and PYSPARK_SUBMIT_ARGS.

buildPySparkShellCommand defines an internal pyargs collection for the parts of the shell command to execute.

buildPySparkShellCommand stores the Python executable (in pyargs) to be the first specified in the following order:

• spark.pyspark.driver.python configuration property
• spark.pyspark.python configuration property
• PYSPARK_DRIVER_PYTHON environment variable
• PYSPARK_PYTHON environment variable
• python3

buildPySparkShellCommand sets the environment variables (for the Python executable to use), if specified.

In the end, buildPySparkShellCommand copies all the options from PYSPARK_DRIVER_PYTHON_OPTS, if specified.

buildSparkSubmitCommand¶

List<String> buildSparkSubmitCommand(
  Map<String, String> env)

buildSparkSubmitCommand starts by building so-called effective config. When in client mode, buildSparkSubmitCommand adds spark.driver.extraClassPath to the result Spark command.

buildSparkSubmitCommand builds the first part of the Java command passing in the extra classpath (only for client deploy mode).

buildSparkSubmitCommand appends SPARK_SUBMIT_OPTS and SPARK_JAVA_OPTS environment variables.

(only for client deploy mode) ...

addPermGenSizeOpt case...elaborate

buildSparkSubmitCommand appends org.apache.spark.deploy.SparkSubmit and the command-line arguments (using buildSparkSubmitArgs).

buildSparkSubmitArgs¶

List<String> buildSparkSubmitArgs()

buildSparkSubmitArgs builds a list of command-line arguments for spark-submit.

buildSparkSubmitArgs uses a SparkSubmitOptionParser to add the command-line arguments that spark-submit recognizes (when it is executed later on and uses the very same SparkSubmitOptionParser parser to parse command-line arguments).

buildSparkSubmitArgs is used when:

• InProcessLauncher is requested to startApplication
• SparkLauncher is requested to createBuilder
• SparkSubmitCommandBuilder is requested to buildSparkSubmitCommand and constructEnvVarArgs

SparkSubmitCommandBuilder Properties and SparkSubmitOptionParser Attributes¶