Skip to content

== [[Utils]] Utils Helper Object

Utils is a Scala object that...FIXME

=== [[getLocalDir]] getLocalDir Method

[source, scala]

getLocalDir(conf: SparkConf): String



getLocalDir is used when:

  • Utils is requested to <>

  • SparkEnv is[created] (on the driver)

  •[spark-shell] is launched

  • Spark on YARN's Client is requested to[prepareLocalResources] and[create archive with configuration files and Spark configuration]

  • PySpark's PythonBroadcast is requested to readObject

* PySpark's EvalPythonExec is requested to doExecute

=== [[fetchFile]] fetchFile Method

[source, scala]

fetchFile( url: String, targetDir: File, conf: SparkConf, securityMgr: SecurityManager, hadoopConf: Configuration, timestamp: Long, useCache: Boolean): File



fetchFile is used when:

  • SparkContext is requested to[addFile]

  • Executor is requested to[updateDependencies]

* Spark Standalone's DriverRunner is requested to downloadUserJar

=== [[getOrCreateLocalRootDirsImpl]] getOrCreateLocalRootDirsImpl Internal Method

[source, scala]

getOrCreateLocalRootDirsImpl(conf: SparkConf): Array[String]


NOTE: getOrCreateLocalRootDirsImpl is used exclusively when Utils is requested to <>

=== [[getOrCreateLocalRootDirs]] getOrCreateLocalRootDirs Internal Method

[source, scala]

getOrCreateLocalRootDirs(conf: SparkConf): Array[String]



getOrCreateLocalRootDirs is used when:

  • Utils is requested to <>

* Worker is requested to[handle a LaunchExecutor message]

Last update: 2020-10-06