Skip to content

DatasetManager

Scala object

DatasetManager is an object in Scala which means it is a class that has exactly one instance (itself). A Scala object is created lazily when it is referenced for the first time.

Learn more in Tour of Scala.

materializeDatasets

materializeDatasets(
  resolvedDataflowGraph: DataflowGraph,
  context: PipelineUpdateContext): DataflowGraph

materializeDatasets...FIXME

materializeDatasets is used when:

constructFullRefreshSet

constructFullRefreshSet(
  graphTables: Seq[Table],
  context: PipelineUpdateContext): (Seq[Table], Seq[TableIdentifier], Seq[TableIdentifier])

constructFullRefreshSet gives the following collections:

  • Tables to be refreshed (incl. a full refresh)
  • TableIdentifiers of the tables to be refreshed (excl. fully refreshed)
  • TableIdentifiers of the tables to be fully refreshed only

If there are tables to be fully refreshed yet not allowed for a full refresh, constructFullRefreshSet prints out the following INFO message to the logs:

Skipping full refresh on some tables because pipelines.reset.allowed was set to false.
Tables: [fullRefreshNotAllowed]

constructFullRefreshSet...FIXME

constructFullRefreshSet is used when: