ReliableCheckpointRDD can be given a Partitioner to be created.
When requested for the Partitioner (as an RDD), ReliableCheckpointRDD returns the one it was created with or reads the partitioner from the given RDD checkpoint directory, if exists.
writeRDDToCheckpointDirectory[T: ClassTag]( originalRDD: RDD[T], checkpointDir: String, blockSize: Int = -1): ReliableCheckpointRDD[T]
writeRDDToCheckpointDirectory is used when ReliableRDDCheckpointData is requested to doCheckpoint.
writePartitionerToCheckpointDir( sc: SparkContext, partitioner: Partitioner, checkpointDirPath: Path): Unit
In the end, writePartitionerToCheckpointDir prints out the following DEBUG message to the logs:
Written partitioner to [partitionerFilePath]
In case of any non-fatal exception, writePartitionerToCheckpointDir prints out the following DEBUG message to the logs:
Error writing partitioner [partitioner] to [checkpointDirPath]
writePartitionerToCheckpointDir is used when ReliableCheckpointRDD is requested to write the RDD to the checkpoint directory.
readCheckpointedPartitionerFile( sc: SparkContext, checkpointDirPath: String): Option[Partitioner]
readCheckpointedPartitionerFile prints out the following DEBUG message to the logs and returns the partitioner.
Read partitioner from [partitionerFilePath]
In case of FileNotFoundException or any non-fatal exceptions, readCheckpointedPartitionerFile prints out a corresponding message to the logs and returns None.
readCheckpointedPartitionerFile is used when ReliableCheckpointRDD is requested for the Partitioner.
ALL logging level for
org.apache.spark.rdd.ReliableCheckpointRDD$ logger to see what happens inside.
Add the following line to
Refer to Logging.