SparkSubmit¶

SparkSubmit is the entry point to spark-submit shell script.

Actions¶

SparkSubmit executes actions (based on the action argument).

Killing Submission¶

kill(
  args: SparkSubmitArguments): Unit

kill...FIXME

Displaying Version¶

printVersion(): Unit

printVersion...FIXME

Submission Status¶

requestStatus(
  args: SparkSubmitArguments): Unit

requestStatus...FIXME

Submission¶

submit(
  args: SparkSubmitArguments,
  uninitLog: Boolean): Unit

submit...FIXME

Running Main Class¶

runMain(
  args: SparkSubmitArguments,
  uninitLog: Boolean): Unit

runMain prepareSubmitEnvironment with the given SparkSubmitArguments (that gives a 4-element tuple of childArgs, childClasspath, sparkConf and childMainClass).

With verbose enabled, runMain prints out the following INFO messages to the logs:

Main class:
[childMainClass]
Arguments:
[childArgs]
Spark config:
[sparkConf_redacted]
Classpath elements:
[childClasspath]

runMain creates and sets a context classloader (based on spark.driver.userClassPathFirst configuration property) and adds the jars (from childClasspath).

runMain loads the main class (childMainClass).

runMain creates a SparkApplication (if the main class is a subtype of) or creates a JavaMainApplication (with the main class).

In the end, runMain requests the SparkApplication to start (with the childArgs and sparkConf).

Cluster Managers¶

SparkSubmit has a built-in support for some cluster managers (that are selected based on the master argument).

Launching Standalone Application¶

main(
  args: Array[String]): Unit

main...FIXME

doSubmit¶

doSubmit(
  args: Array[String]): Unit

doSubmit...FIXME

doSubmit is used when:

• InProcessSparkSubmit standalone application is started
• SparkSubmit standalone application is started

prepareSubmitEnvironment¶

prepareSubmitEnvironment(
  args: SparkSubmitArguments,
  conf: Option[HadoopConfiguration] = None): (Seq[String], Seq[String], SparkConf, String)

prepareSubmitEnvironment creates a 4-element tuple made up of the following:

1. childArgs for arguments
2. childClasspath for Classpath elements
3. sysProps for Spark properties
4. childMainClass

prepareSubmitEnvironment...FIXME

prepareSubmitEnvironment determines the cluster manager based on master argument.

For KUBERNETES, prepareSubmitEnvironment checkAndGetK8sMasterUrl.

prepareSubmitEnvironment...FIXME

prepareSubmitEnvironment is used when...FIXME

childMainClass¶

childMainClass is the last 4^th argument in the result tuple of prepareSubmitEnvironment.

// (childArgs, childClasspath, sparkConf, childMainClass)
(Seq[String], Seq[String], SparkConf, String)

childMainClass can be as follows:

isKubernetesClient¶

prepareSubmitEnvironment uses isKubernetesClient flag to indicate that:

• Cluster manager is Kubernetes
• Deploy mode is client

isKubernetesClusterModeDriver¶

prepareSubmitEnvironment uses isKubernetesClusterModeDriver flag to indicate that:

• isKubernetesClient
• spark.kubernetes.submitInDriver configuration property is enabled (Spark on Kubernetes)

renameResourcesToLocalFS¶

renameResourcesToLocalFS(
  resources: String,
  localResources: String): String

renameResourcesToLocalFS...FIXME

renameResourcesToLocalFS is used for isKubernetesClusterModeDriver mode.

downloadResource¶

downloadResource(
  resource: String): String

downloadResource...FIXME

Checking Whether Resource is Internal¶

isInternal(
  res: String): Boolean

isInternal is true when the given res is spark-internal.

isInternal is used when:

• SparkSubmit is requested to isUserJar
• SparkSubmitArguments is requested to handleUnknown

Checking Whether Resource is User Jar¶

isUserJar(
  res: String): Boolean

isUserJar is true when the given res is none of the following:

• isShell
• isPython
• isInternal
• isR

isUserJar is used when:

• FIXME