Day 8: dbt Fundamentals Course (part 2)¶
This is Part 2 of the notes from the dbt Fundamentals course.
Cloud Data Warehouses and ELT¶
Traditional Data Teams (ETL projects) consist of the following roles:
- Data Engineers who maintains data infrastructure and the ETL process for creating tables and views
- Data Analysts focus on querying tables and views to drive business insights for stakeholders
Cloud Data Warehouses / Data Lakes made ELT and a new role at the forefront of data projects:
- Analytics Engineer who transforms raw data (through layers) up to the BI layer
- Think of Databricks / Delta Lake's Medallion Architecture
- Think of bronze, silver and gold tables
- In charge of
T
in ELT- Data Engineer can focus on the EL part of ELT (i.e. extracting data from sources and loading data into data warehouse) and maintaining data infrastructure
- Data Analyst works closely with Analytics Engineer to deliver final (gold) tables that can be queried by BI tools (for faster business decisions)
ELT Influencers¶
- Modern cloud-based data warehouses with scalable storage and compute
- Many data pipeline/extraction tools
- Self-service business intelligence tools increased the ability for stakeholders to access and analyze data
Modern Data Team¶
Modern Data Team (ELT projects) consists of the following roles:
- Data Engineer
- Analytics Engineer
- Data Analyst
dbt Workflow¶
- Firstly, extract and load raw data into a data warehouse using EL tools (loaders)
- Data warehouse could be Snowflake, Redshift, BigQuery or Databricks
- these were mentioned explicitly
- dbt transformations create gold tables for BI tools, ML models, Operational Analytics
- Only
select
statements- No need for DDLs, DMLs or anything but
select
s
- No need for DDLs, DMLs or anything but
- dbt DAG for data lineage
dbt Cloud¶
- Welcome to dbt Cloud!
- Data Warehouses listed explicitly while setting up the first Analytics project
- PostgreSQL
- Redshift
- Snowflake
- BigQuery
- Apache Spark
- Databricks
Summary¶
The Module 2 finishes with such an exceptionally compehensive Review that it'd be a shame to copy it here. Go and read it yourself!