FileNames Utility¶

Create Staged Commit Directory for Log Path¶

commitDirPath(
  logPath: Path): Path

commitDirPath creates a new Path (Apache Hadoop) for the staged commit directory under the given logPath.

commitDirPath is used when:

• DeltaLog is requested to createLogDirectoriesIfNotExists
• MetadataCleanup is requested to cleanUpExpiredLogs
• Snapshot is requested to assertLogFilesBelongToTable
• VacuumCommandImpl is requested to getFilesFromDeltaLog
• FileNames is requested to unbackfilledDeltaFile

Unbackfilled Delta File¶

unbackfilledDeltaFile(
  logPath: Path,
  version: Long,
  uuidString: Option[String] = None): Path

unbackfilledDeltaFile creates a path of the following format:

[logPath]/_staged_commits/[version].[uuid].json

unbackfilledDeltaFile creates a staged commit directory for the given logPath.

unbackfilledDeltaFile creates a new Path (Apache Hadoop) for [version].[uuid].json in the staged commit directory.

unbackfilledDeltaFile is used when:

• DeltaCommitFileProvider is requested to deltaFile and listSortedUnbackfilledDeltaFiles

Staged Commit Directory¶

FileNames defines _staged_commits directory for staged commits.

Used when:

• FileNames is requested to commitDirPath
• BackfilledDeltaFile is requested to extract the version from a Path
• DeltaFile extractor is requested to unapply
• UnbackfilledDeltaFile extractor is requested to extract the version and UUID from a Path

BackfilledDeltaFile¶

Extract Version from FileStatus¶

unapply(
  f: FileStatus): Option[(FileStatus, Long)]

unapply...FIXME

unapply is used when:

• FIXME

Extract Version from Path¶

unapply(
  path: Path): Option[(Path, Long)]

unapply...FIXME

unapply is used when:

• FIXME

UnbackfilledDeltaFile¶

Extract FileStatus Metadata¶

unapply(
  f: FileStatus): Option[(FileStatus, Long, String)]

unapply destructures the path of the given FileStatus (Apache Hadoop).

If successful, unapply returns a tuple of three elements:

1. The given FileStatus
2. version
3. uuidString

unapply is used when:

• MetadataCleanup is requested to cleanUpExpiredLogs
• VacuumCommandImpl is requested to getFilesFromDeltaLog
• CoordinatedCommitsUtils is requested to backfillWhenCoordinatedCommitsDisabled, commitFilesIterator, unbackfilledCommitsPresent
• DeltaCommitFileProvider is requested to apply

Extract Path Metadata¶

unapply(
  path: Path): Option[(Path, Long, String)]

unapply checks if the parent directory of the given Path (Apache Hadoop) is _staged_commits.

If so, unapply destructures the given path (using uuidDeltaFileRegex regexp) into a tuple of three elements:

1. The given Path
2. version
3. uuidString

Otherwise, unapply is None.

unapply is used when:

• FileNames is requested to isUnbackfilledDeltaFile and destructure FileStatus

Creating Hadoop Path To Delta Commit File¶

deltaFile(
  path: Path,
  version: Long): Path

deltaFile creates a Path (Apache Hadoop) to a file in the path directory.

The format of the file is as follows:

[version with leading 0s, up to 20 digits].json

Examples:

• 00000000000000000001.json
• 00000000000000012345.json

deltaFile is used when:

• DeltaHistoryManager is requested for the history of a delta table (and getCommitInfo)
• OptimisticTransactionImpl is requested to commit large and to commit (and write a commit file)
• SnapshotManagement is requested for the LogSegment for a given version (and validateDeltaVersions)
• DESCRIBE DETAIL command is executed (and describeDeltaTable)

Creating Hadoop Path To Compacted Delta File¶

compactedDeltaFile(
  path: Path,
  fromVersion: Long,
  toVersion: Long): Path

compactedDeltaFile creates a Path (Apache Hadoop) to a file in the path directory.

The format of the file is as follows:

[fromVersion with leading 0s, up to 20 digits].[toVersion with leading 0s, up to 20 digits].compacted.json

Examples:

• 00000000000000000001.00000000000000012345.compacted.json