Skip to content

Day 1: Installing dbt

pyenv

$ pyenv --version
pyenv 2.3.2

$ pyenv install 3.10:latest
Downloading Python-3.10.5.tar.xz...
-> https://www.python.org/ftp/python/3.10.5/Python-3.10.5.tar.xz
Installing Python-3.10.5...
Installed Python-3.10.5 to /Users/jacek/.pyenv/versions/3.10.5

$ pyenv rehash

$ pyenv local 3.10.5

$ python --version
Python 3.10.5

$ python3.10 --version
Python 3.10.5

$ pyenv version
3.10.5 (set by /Users/jacek/dev/sandbox/dbt-sandbox/.python-version)

pyenv-virtualenv

pyenv-virtualenv is a pyenv plugin for managing virtualenvs and conda environments for Python on UNIX-like systems.

$ pyenv virtualenv dbt

$ pyenv virtualenvs --skip-aliases
dbt (created from /usr/local/opt/python@3.9)

$ pyenv local dbt

$ cat .python-version
dbt

Install dbt (incl. Databricks adapter plugin)

This is a modified version of the official Install dbt from source to avoid polluting system-wide configuration.

$ pip install --upgrade pip wheel setuptools
Successfully installed pip-22.1.2 setuptools-62.6.0 wheel-0.37.1

$ cat requirements.txt
dbt-core
dbt-postgres
dbt-databricks

$ pip install -r requirements.txt
...
Successfully installed Babel-2.10.3 Jinja2-2.11.3 MarkupSafe-2.0.1 agate-1.6.3 attrs-21.4.0 certifi-2022.6.15 cffi-1.15.1 charset-normalizer-2.1.0 click-8.1.3 colorama-0.4.4 databricks-sql-connector-2.0.2 dbt-core-1.1.1 dbt-databricks-1.1.0 dbt-extractor-0.4.1 dbt-postgres-1.1.1 dbt-spark-1.1.0 future-0.18.2 hologram-0.0.15 idna-3.3 isodate-0.6.1 jsonschema-3.2.0 leather-0.3.4 logbook-1.5.3 mashumaro-2.9 minimal-snowplow-tracker-0.0.2 msgpack-1.0.4 networkx-2.8.3 numpy-1.23.0 packaging-21.3 pandas-1.4.3 parsedatetime-2.4 psycopg2-binary-2.9.3 pyarrow-8.0.0 pycparser-2.21 pyparsing-3.0.9 pyrsistent-0.18.1 python-dateutil-2.8.2 python-slugify-6.1.2 pytimeparse-1.1.8 pytz-2022.1 pyyaml-6.0 requests-2.28.1 six-1.16.0 sqlparams-4.0.0 sqlparse-0.4.2 text-unidecode-1.3 thrift-0.16.0 typing-extensions-4.3.0 urllib3-1.26.9 werkzeug-2.1.2

$ pip list dbt | egrep "^dbt-" | sort
dbt-core                 1.1.1
dbt-databricks           1.1.0
dbt-extractor            0.4.1
dbt-postgres             1.1.1
dbt-spark                1.1.0

Install from Sources

Clone the dbt-databricks adapter from https://github.com/databricks/dbt-databricks and install from the directory.

git clone https://github.com/databricks/dbt-databricks
cd dbt-databricks
gco v1.1.0
gswc v1.1.0
pip install .
Successfully installed dbt-databricks-1.1.0