CoordinatedCommitsUtils¶

getCommitsFromCommitCoordinatorWithUsageLogs¶

getCommitsFromCommitCoordinatorWithUsageLogs(
  deltaLog: DeltaLog,
  tableCommitCoordinatorClient: TableCommitCoordinatorClient,
  catalogTableOpt: Option[CatalogTable],
  startVersion: Long,
  versionToLoad: Option[Long],
  isAsyncRequest: Boolean): GetCommitsResponse

getCommitsFromCommitCoordinatorWithUsageLogs...FIXME

getCommitsFromCommitCoordinatorWithUsageLogs is used when:

• SnapshotManagement is requested to listDeltaCompactedDeltaCheckpointFilesAndLatestChecksumFile

commitFilesIterator¶

commitFilesIterator(
  deltaLog: DeltaLog,
  catalogTableOpt: Option[CatalogTable],
  startVersion: Long): Iterator[(FileStatus, Long)]

commitFilesIterator...FIXME

commitFilesIterator is used when:

• DeltaLog is requested to getChangeLogFiles

getCommitCoordinatorClient¶

getCommitCoordinatorClient(
  spark: SparkSession,
  deltaLog: DeltaLog, // Used for logging
  metadata: Metadata,
  protocol: Protocol,
  failIfImplUnavailable: Boolean): Option[CommitCoordinatorClient]

getCommitCoordinatorClient...FIXME

getCommitCoordinatorClient is used when:

• OptimisticTransactionImpl is requested to registerTableForCoordinatedCommitsIfNeeded
• CoordinatedCommitsUtils is requested to getTableCommitCoordinator

getTableCommitCoordinator¶

getTableCommitCoordinator(
      spark: SparkSession,
      deltaLog: DeltaLog, // Used for logging
      snapshotDescriptor: SnapshotDescriptor,
      failIfImplUnavailable: Boolean): Option[TableCommitCoordinatorClient]

getTableCommitCoordinator...FIXME

getTableCommitCoordinator is used when:

• Snapshot is requested for a TableCommitCoordinatorClient

getCoordinatedCommitsConfs¶

getCoordinatedCommitsConfs(
  metadata: Metadata): (Option[String], Map[String, String])

getCoordinatedCommitsConfs...FIXME

getCoordinatedCommitsConfs is used when:

• OptimisticTransactionImpl is requested to registerTableForCoordinatedCommitsIfNeeded

unbackfilledCommitsPresent¶

unbackfilledCommitsPresent(
  snapshot: Snapshot): Boolean

unbackfilledCommitsPresent is enabled (true) when there are unbackfilled delta files (files in _staged_commits directory).

unbackfilledCommitsPresent is used when:

• CoordinatedCommitsPreDowngradeCommand is requested to removeFeatureTracesIfNeeded
• CatalogOwnedTableFeature is requested to validateDropInvariants
• CoordinatedCommitsTableFeature is requested to validateDropInvariants

validateConfigurationsForCreateDeltaTableCommand¶

validateConfigurationsForCreateDeltaTableCommand(
  spark: SparkSession,
  tableExists: Boolean,
  query: Option[LogicalPlan],
  catalogTableProperties: Map[String, String]): Unit

validateConfigurationsForCreateDeltaTableCommand validateConfigurationsForCreateDeltaTableCommandImpl.

validateConfigurationsForCreateDeltaTableCommand is used when:

• CreateDeltaTableCommand is requested to run

validateConfigurationsForCreateDeltaTableCommandImpl¶

validateConfigurationsForCreateDeltaTableCommandImpl(
  spark: SparkSession,
  propertyOverrides: Map[String, String],
  tableExists: Boolean,
  command: String): Unit

validateConfigurationsForCreateDeltaTableCommandImpl...FIXME

verifyContainsOnlyCoordinatorNameAndConf¶

verifyContainsOnlyCoordinatorNameAndConf(
  properties: Map[String, String],
  command: String,
  fromDefault: Boolean): Unit

verifyContainsOnlyCoordinatorNameAndConf...FIXME

verifyContainsOnlyCoordinatorNameAndConf is used when:

• CoordinatedCommitsUtils is requested to validateConfigurationsForAlterTableSetPropertiesDeltaCommand and validateConfigurationsForCreateDeltaTableCommandImpl

getExplicitCCConfigurations¶

getExplicitCCConfigurations(
  properties: Map[String, String]): Map[String, String]

getExplicitCCConfigurations extracts the Coordinated Commits-related entries from the given properties map.

getExplicitCCConfigurations is used when:

• OptimisticTransactionImpl is requested to updateMetadataForNewTableInReplace
• CloneTableBase is requested to determineCoordinatedCommitsConfigurations and determineTargetMetadata
• CreateDeltaTableCommand is requested to filterCoordinatedCommitsProperties
• CoordinatedCommitsUtils is requested to validateConfigurationsForAlterTableSetPropertiesDeltaCommand, validateConfigurationsForAlterTableUnsetPropertiesDeltaCommand and validateConfigurationsForCreateDeltaTableCommandImpl

TABLE_PROPERTY_CONFS¶

TABLE_PROPERTY_CONFS is a collection of the following DeltaConfigs:

• delta.coordinatedCommits.commitCoordinator-preview
• delta.coordinatedCommits.commitCoordinatorConf-preview
• delta.coordinatedCommits.tableConf-preview

TABLE_PROPERTY_CONFS is used when:

• CoordinatedCommitsUtils is requested to getDefaultCCConfigurations and TABLE_PROPERTY_KEYS

TABLE_PROPERTY_KEYS¶

TABLE_PROPERTY_KEYS: Seq[String]

TABLE_PROPERTY_KEYS is a collection of the keys of the delta table properties:

• delta.coordinatedCommits.commitCoordinator-preview
• delta.coordinatedCommits.commitCoordinatorConf-preview
• delta.coordinatedCommits.tableConf-preview

TABLE_PROPERTY_KEYS is used when:

• DeltaAnalysis is requested to apply
• OptimisticTransactionImpl is requested to updateMetadataForNewTableInReplace
• CoordinatedCommitsPreDowngradeCommand is requested to removeFeatureTracesIfNeeded
• CloneTableBase is requested to prepareSourceMetadata
• CreateDeltaTableCommand is requested to filterCoordinatedCommitsProperties
• CoordinatedCommitsUtils is requested to getExplicitCCConfigurations, tablePropertiesPresent and validateConfigurationsForAlterTableUnsetPropertiesDeltaCommand

tablePropertiesPresent¶

tablePropertiesPresent(
  metadata: Metadata): Boolean

tablePropertiesPresent checks if any of the TABLE_PROPERTY_KEYS is among the table properties in the configuration of the given Metadata.

tablePropertiesPresent is used when:

• CoordinatedCommitsPreDowngradeCommand is requested to removeFeatureTracesIfNeeded
• CoordinatedCommitsTableFeature is requested to validateDropInvariants