Dataflow Graph¶
Dataflow Graph (DataflowGraph) is a graph of tables, sinks, views and flows in a Spark Declarative Pipelines project.
A Dataflow Graph is created every time Spark Pipelines CLI (spark-pipelines) is launched to run a pipeline update (with either run or dry-run option).
Spark Pipelines CLI uses Spark Connect to talk to PipelinesHandler to run a pipeline update.
PipelinesHandler prints out the following INFO message to the logs:
PipelinesHandler: Start pipeline cmd received: start_run {
dataflow_graph_id: "..."
full_refresh_all: ...
dry: ...
storage: "..."
}
PipelinesHandler uses GraphRegistrationContext to register GraphElements and, eventually, build a DataflowGraph.