Skip to content

DependencyUtils Utilities

resolveGlobPaths

resolveGlobPaths(
  paths: String,
  hadoopConf: Configuration): String

resolveGlobPaths...FIXME

resolveGlobPaths is used when:

downloadFile

downloadFile(
  path: String,
  targetDir: File,
  sparkConf: SparkConf,
  hadoopConf: Configuration,
  secMgr: SecurityManager): String

downloadFile resolves the path to a well-formed URI and branches off based on the scheme:

  • For file and local schemes, downloadFile returns the input path
  • For other schemes, downloadFile...FIXME

downloadFile is used when:

downloadFileList

downloadFileList(
  fileList: String,
  targetDir: File,
  sparkConf: SparkConf,
  hadoopConf: Configuration,
  secMgr: SecurityManager): String

downloadFileList...FIXME

downloadFileList is used when:

resolveMavenDependencies

resolveMavenDependencies(
  packagesExclusions: String,
  packages: String,
  repositories: String,
  ivyRepoPath: String,
  ivySettingsPath: Option[String]): String

resolveMavenDependencies...FIXME

resolveMavenDependencies is used when:

  • SparkSubmit is requested to prepareSubmitEnvironment (for all resource managers but Spark Standalone and Apache Mesos)

Adding Local Jars to ClassLoader

addJarToClasspath(
  localJar: String,
  loader: MutableURLClassLoader): Unit

addJarToClasspath adds file and local jars (as localJar) to the loader classloader.

addJarToClasspath resolves the URI of localJar. If the URI is file or local and the file denoted by localJar exists, localJar is added to loader. Otherwise, the following warning is printed out to the logs:

Warning: Local jar /path/to/fake.jar does not exist, skipping.

For all other URIs, the following warning is printed out to the logs:

Warning: Skip remote jar hdfs://fake.jar.

Note

addJarToClasspath assumes file URI when localJar has no URI specified, e.g. /path/to/local.jar.

resolveAndDownloadJars

resolveAndDownloadJars(
  jars: String,
  userJar: String,
  sparkConf: SparkConf,
  hadoopConf: Configuration,
  secMgr: SecurityManager): String

resolveAndDownloadJars...FIXME

resolveAndDownloadJars is used when:

  • DriverWrapper is requested to setupDependencies (Spark Standalone cluster mode)