Skip to content

DriverLogger

DriverLogger runs on the driver (in client deploy mode) to copy driver logs to Hadoop DFS periodically.

Creating Instance

DriverLogger takes the following to be created:

DriverLogger is created using apply utility.

Creating DriverLogger

apply(
  conf: SparkConf): Option[DriverLogger]

apply creates a DriverLogger when the following hold:

  1. spark.driver.log.persistToDfs.enabled configuration property is enabled
  2. The Spark application runs in client deploy mode (and spark.submit.deployMode is client)
  3. spark.driver.log.dfsDir is specified

apply prints out the following WARN message to the logs with no spark.driver.log.dfsDir specified:

Driver logs are not persisted because spark.driver.log.dfsDir is not configured

apply is used when:

Starting DfsAsyncWriter

startSync(
  hadoopConf: Configuration): Unit

startSync creates and starts a DfsAsyncWriter (with the spark.app.id configuration property).

startSync is used when: