HadoopMapRedWriteConfigUtil¶

HadoopMapRedWriteConfigUtil is a HadoopWriteConfigUtil for RDD.saveAsHadoopDataset operator.

Creating Instance¶

HadoopMapRedWriteConfigUtil takes the following to be created:

• SerializableJobConf

HadoopMapRedWriteConfigUtil is created when:

• PairRDDFunctions is requested to saveAsHadoopDataset

Logging¶

Enable ALL logging level for org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil logger to see what happens inside.

Add the following line to conf/log4j2.properties:

logger.HadoopMapRedWriteConfigUtil.name = org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil
logger.HadoopMapRedWriteConfigUtil.level = all

Refer to Logging.