Skip to content


DriverLogger runs on the driver (in client deploy mode) to copy driver logs to Hadoop DFS periodically.

Creating Instance

DriverLogger takes the following to be created:

DriverLogger is created using apply utility.

Creating DriverLogger

  conf: SparkConf): Option[DriverLogger]

apply creates a DriverLogger when the following hold:

  1. spark.driver.log.persistToDfs.enabled configuration property is enabled
  2. The Spark application runs in client deploy mode (and spark.submit.deployMode is client)
  3. spark.driver.log.dfsDir is specified

apply prints out the following WARN message to the logs with no spark.driver.log.dfsDir specified:

Driver logs are not persisted because spark.driver.log.dfsDir is not configured

apply is used when:

Starting DfsAsyncWriter

  hadoopConf: Configuration): Unit

startSync creates and starts a DfsAsyncWriter (with the configuration property).

startSync is used when: