Skip to content

HadoopWriteConfigUtil

HadoopWriteConfigUtil[K, V] is an abstraction of writer configurers for SparkHadoopWriter to write a key-value RDD (for RDD.saveAsNewAPIHadoopDataset and RDD.saveAsHadoopDataset operators).

Contract

assertConf

assertConf(
  jobContext: JobContext,
  conf: SparkConf): Unit

closeWriter

closeWriter(
  taskContext: TaskAttemptContext): Unit

createCommitter

createCommitter(
  jobId: Int): HadoopMapReduceCommitProtocol

Creates a HadoopMapReduceCommitProtocol committer

Used when:

createJobContext

createJobContext(
  jobTrackerId: String,
  jobId: Int): JobContext

createTaskAttemptContext

createTaskAttemptContext(
  jobTrackerId: String,
  jobId: Int,
  splitId: Int,
  taskAttemptId: Int): TaskAttemptContext

Creates a Hadoop TaskAttemptContext

initOutputFormat

initOutputFormat(
  jobContext: JobContext): Unit

initWriter

initWriter(
  taskContext: TaskAttemptContext,
  splitId: Int): Unit

write

write(
  pair: (K, V)): Unit

Writes out the key-value pair

Used when:

Implementations