HadoopMapReduceWriteConfigUtil¶

HadoopMapReduceWriteConfigUtil is a HadoopWriteConfigUtil for RDD.saveAsNewAPIHadoopDataset operator.

Creating Instance¶

HadoopMapReduceWriteConfigUtil takes the following to be created:

• SerializableConfiguration

HadoopMapReduceWriteConfigUtil is created when:

• PairRDDFunctions is requested to saveAsNewAPIHadoopDataset

Logging¶

Enable ALL logging level for org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil logger to see what happens inside.

Add the following line to conf/log4j2.properties:

logger.HadoopMapReduceWriteConfigUtil.name = org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil
logger.HadoopMapReduceWriteConfigUtil.level = all

Refer to Logging.