LocalRDDCheckpointData is a RDDCheckpointData.
LocalRDDCheckpointData takes the following to be created:
LocalRDDCheckpointData is created when:
RDDis requested to localCheckpoint
doCheckpoint is part of the RDDCheckpointData abstraction.
doCheckpoint creates a LocalCheckpointRDD with the RDD.
doCheckpoint triggers caching any missing partitions (by checking availability of the RDDBlockIds for the partitions in the BlockManagerMaster).
Extra Spark Job
If there are any missing partitions (
doCheckpoint requests the
SparkContext to run a Spark job with the
RDD and the missing partitions.
doCheckpointmakes sure that the StorageLevel of the
RDD uses disk (among other persistence storages). If not,
doCheckpoint throws an
Storage level [level] is not appropriate for local checkpointing