Data Build Tool (dbt )

 

dbt (data build tool) is a metadata-driven transformation framework that functions as a DAG-based SQL compiler and execution orchestrator for cloud data warehouses. Internally, it parses project files to construct a dependency graph using ref() and source(), then compiles Jinja-templated models into optimized SQL via its macro engine. Execution is delegated to the warehouse, with parallelization governed by graph topology. Core artifacts like manifest.json encode full lineage, configurations, and compiled nodes, while run_results.json captures execution telemetry. This architecture positions dbt as a control plane that unifies transformation logic, lineage, testing, and observability within modern data platforms.

What dbt Really Is (Architect Perspective)

At its core, dbt is a:

👉 Metadata-driven transformation framework
👉 SQL compiler + DAG execution engine
👉 Control plane over warehouse compute

Inside dbt Internals

• DAG
• Manifest.json
• Execution Engine

dbt is NOT a processing engine

• SQL Compiler + DAG Execution Framework

DAG Parsing

• dbt scans project files
• Builds dependency graph using ref()
• Creates Directed Acyclic Graph

Graph Structure

Each node =

• Model
• Test
• Seed

Each edge = dependency

👉 This drives execution order

manifest.json

The Brain of dbt

Contains:

• DAG structure
• Model metadata
• Compiled SQL
• Lineage

Why manifest.json Matters

• Powers dbt docs
• Enables lineage tools
• Integrates with DataHub / OpenLineage

Compilation Engine

Jinja SQL → Compiled SQL

Includes:

• Macros
• Variables
• Environment configs

Execution Model

dbt:
❌ Does NOT process data
✅ Pushes SQL to warehouse

Parallel execution based on DAG

run_results.json

Tracks:

• Execution status
• Runtime metrics
• Failures

👉 Used for observability

Architect Insight

If you understand:
✔ DAG
✔ manifest.json

👉 You understand dbt at scale

dbt = Metadata-driven transformation layer

Core vs Cloud vs Fusion — Strategic Comparison