FileContextBasedCheckpointFileManager¶

FileContextBasedCheckpointFileManager is a CheckpointFileManager that uses FileContext (Apache Hadoop).

FileContextBasedCheckpointFileManager is the default CheckpointFileManager (unless spark.sql.streaming.checkpointFileManagerClass is defined) as HDFS's FileContext.rename() gives atomic renames, which is used for createAtomic.

Creating Instance¶

FileContextBasedCheckpointFileManager takes the following to be created:

• Path (Apache Hadoop)
• Configuration (Apache Hadoop)

FileContextBasedCheckpointFileManager is created when:

• CheckpointFileManager is requested to create a CheckpointFileManager (and spark.sql.streaming.checkpointFileManagerClass is not defined)

createAtomic¶

createAtomic(
  path: Path,
  overwriteIfPossible: Boolean): CancellableFSDataOutputStream

createAtomic is part of the CheckpointFileManager abstraction.

createAtomic creates a RenameBasedFSDataOutputStream.