Skip to content

Dataflow Graph

Dataflow Graph (DataflowGraph) is a graph of tables, sinks, views and flows in a Spark Declarative Pipelines project.

A Dataflow Graph is created every time Spark Pipelines CLI (spark-pipelines) is launched to run a pipeline update (with either run or dry-run option).

Spark Pipelines CLI uses Spark Connect to talk to PipelinesHandler to run a pipeline update.

PipelinesHandler prints out the following INFO message to the logs:

PipelinesHandler: Start pipeline cmd received: start_run {
  dataflow_graph_id: "..."
  full_refresh_all: ...
  dry: ...
  storage: "..."
}

PipelinesHandler uses GraphRegistrationContext to register GraphElements and, eventually, build a DataflowGraph.