Create dbt-MariaDB Virtual Environment

About this task

Required only when MariaDB is the CDM target database. The IF02 DAGs activate ~/mariadb/bin/activate before running dbt for MariaDB.

Procedure

  1. Create a virtual environment for MariaDB:
    python3 -m venv ~/mariadb
  2. Upgrade pip and install dbt Core, the MariaDB dbt adapter, and PyMySQL in the virtual environment:
    ~/mariadb/bin/pip install --upgrade pip
    ~/mariadb/bin/pip install --no-cache-dir dbt-core==<version> \
        <MariaDB dbt adapter>==<version> pymysql
    Note:
    The IF02 package does not specify the dbt-core version or MariaDB dbt adapter package. Obtain both from HCL before installing, and use the same versions on every Airflow worker.
  3. Verify the installation:
    ~/mariadb/bin/dbt --version

What to do next

Airflow Python environment. The ddl_execution_26.1_IF02 DAG runs the DDL modules with the Python interpreter of Airflow, not unica_venv. Install the packages required by the databases you use in that environment. The module/requirements.txt file lists oracledb, snowflake-connector-python, and pyodbc. For MariaDB, install pymysql separately; it is not included in that file.

pip install -r <AirflowDags>/module/requirements.txt
pip install pymysql  # MariaDB only