DriverLogger runs on the driver (in
client deploy mode) to copy driver logs to Hadoop DFS periodically.
DriverLogger takes the following to be created:
DriverLogger is created using apply utility.
apply( conf: SparkConf): Option[DriverLogger]
apply creates a DriverLogger when the following hold:
- spark.driver.log.persistToDfs.enabled configuration property is enabled
- The Spark application runs in
clientdeploy mode (and spark.submit.deployMode is
- spark.driver.log.dfsDir is specified
apply prints out the following WARN message to the logs with no spark.driver.log.dfsDir specified:
Driver logs are not persisted because spark.driver.log.dfsDir is not configured
apply is used when:
startSync( hadoopConf: Configuration): Unit
startSync creates and starts a
DfsAsyncWriter (with the spark.app.id configuration property).
startSync is used when:
SparkContextis requested to postApplicationStart