HadoopMapReduceCommitProtocol¶

HadoopMapReduceCommitProtocol is a FileCommitProtocol.

HadoopMapReduceCommitProtocol is a Serializable (Java) (to be sent out in tasks over the wire to executors).

Creating Instance¶

HadoopMapReduceCommitProtocol takes the following to be created:

• Job ID
• Path
• dynamicPartitionOverwrite flag (default: false)

HadoopMapReduceCommitProtocol is created when:

• HadoopWriteConfigUtil is requested to create a committer
• HadoopMapReduceWriteConfigUtil is requested to create a committer
• HadoopMapRedWriteConfigUtil is requested to create a committer

Logging¶

Enable ALL logging level for org.apache.spark.internal.io.HadoopMapReduceCommitProtocol logger to see what happens inside.

Add the following line to conf/log4j2.properties:

logger.HadoopMapReduceCommitProtocol.name = org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
logger.HadoopMapReduceCommitProtocol.level = all

Refer to Logging.