Skip to content


== [[SparkSubmitArguments]] SparkSubmitArguments -- spark-submit's Command-Line Argument Parser

SparkSubmitArguments is a custom SparkSubmitArgumentsParser to <> the command-line arguments of[spark-submit script] that the[actions] (i.e.[submit],[kill] and[status]) use for their execution (possibly with the explicit env environment).

NOTE: SparkSubmitArguments is created when <spark-submit script>> with only args passed in and later used for printing the arguments in <>.

=== [[loadEnvironmentArguments]] Calculating Spark Properties -- loadEnvironmentArguments internal method

[source, scala]

loadEnvironmentArguments(): Unit

loadEnvironmentArguments calculates the Spark properties for the current execution of[spark-submit].

loadEnvironmentArguments reads command-line options first followed by Spark properties and System's environment variables.

NOTE: Spark config properties start with spark. prefix and can be set using --conf [key=value] command-line option.

=== [[handle]] handle Method

[source, scala]

protected def handle(opt: String, value: String): Boolean

handle parses the input opt argument and returns true or throws an IllegalArgumentException when it finds an unknown opt.

handle sets the internal properties in the table[Command-Line Options, Spark Properties and Environment Variables].

=== [[mergeDefaultSparkProperties]] mergeDefaultSparkProperties Internal Method

[source, scala]

mergeDefaultSparkProperties(): Unit

mergeDefaultSparkProperties merges Spark properties from the[default Spark properties file, i.e. spark-defaults.conf] with those specified through --conf command-line option.

Last update: 2020-10-06