Skip to content

DeltaSourceMetadataTrackingLog

DeltaSourceMetadataTrackingLog is used by DeltaDataSource to track data schema evolution of a delta table.

Creating Instance

DeltaSourceMetadataTrackingLog takes the following to be created:

  • SparkSession (Spark SQL)
  • Metadata Location
  • Snapshot
  • Metadata Path
  • initMetadataLogEagerly flag (default: true)

DeltaSourceMetadataTrackingLog is created using DeltaSourceMetadataTrackingLog.create utility.

Creating DeltaSourceMetadataTrackingLog

create(
  sparkSession: SparkSession,
  rootMetadataLocation: String,
  sourceSnapshot: Snapshot,
  sourceTrackingId: Option[String] = None,
  sourceMetadataPathOpt: Option[String] = None,
  mergeConsecutiveSchemaChanges: Boolean = false,
  initMetadataLogEagerly: Boolean = true): DeltaSourceMetadataTrackingLog

create...FIXME

create is used when: