Lakehouse architecture
dbt on Spark and Trino with Apache Iceberg: building medallion pipelines that stay fast and correct
DataNature runs dbt against three targets: Spark 4 Connect for heavy transformations into Iceberg, Trino for lighter SQL and federated sources, and DN Warehouse for low-latency serving tables. Projects live in Git, follow a staging, silver, intermediate and marts layout, de-duplicate in Silver, use incremental models for large facts, and commit packages so scheduled jobs run offline.
Published . Last updated
On this page
dbt is the transformation layer most data teams already know. On a lakehouse it gains a choice it does not have on a single warehouse: which engine runs each model. This guide covers how dbt works in DataNature, how to pick a target per model, and the habits that keep scheduled pipelines fast and correct.
Which engines can a dbt model run on?
| Target | Engine | Writes to | Use for |
|---|---|---|---|
spark | Spark 4 Connect | lakehouse.<schema> (Iceberg) | Heavy transforms; the default for new projects and jobs |
trino | Trino | lakehouse.<schema> (Iceberg) | Lighter SQL, and models that read federated sources such as Postgres or the warehouse |
dnw | DN Warehouse (serving compute) | Warehouse database | Serving-layer tables in models/warehouse/ |
Use one engine per model. Spark-only configuration such as file_format='iceberg' fails on Trino with INVALID_TABLE_PROPERTY, and models that read relational catalogs fail on Spark with Cannot set database in spark!.
Always pass the target in the terminal. A bare dbt run uses the profiles.yml default, which is Trino:
dbt run --target spark --select silver.orders_clean+
dbt build --target trino --select tag:finance
How should the project be laid out?
models/
staging/ # thin views over lakehouse.bronze_<src>.*: rename, cast
silver/ # cleaned, de-duplicated, conformed entities
intermediate/ # reusable joins
marts/ # gold: facts and dimensions per domain (core/, finance/, ...)
warehouse/ # dnw-only models (build with --target dnw)
snapshots/ # SCD2 history
tests/ macros/ seeds/
With the default profile schema default, +schema: silver builds into lakehouse.default_silver and +schema: gold into lakehouse.default_gold. Declare Bronze tables as sources so lineage and freshness work:
sources:
- name: bronze_crm
database: lakehouse
schema: bronze_crm
tables:
- name: customers
loaded_at_field: last_modified
Where should de-duplication and quality checks happen?
In Silver. Bronze is nullable and append-friendly by design, so Silver is where the data becomes trustworthy. Keep the latest row per key:
select * from (
select *,
row_number() over (partition by id order by last_modified desc) as rn
from {{ source('bronze_crm', 'customers') }}
) where rn = 1
Then enforce it with tests: not_null and unique on Silver keys, and relationships between facts and dimensions in Gold.
How do you keep large models fast?
- Use incremental models for large facts:
materialized='incremental', aunique_key, and a filter on the load timestamp. Full rebuilds of big tables are the most common cause of slow jobs. - Pre-aggregate in Gold whatever dashboards and cubes need. Cubes sit on Gold, never on Bronze.
- Avoid one-to-many joins in cubes; they fan out and inflate sums. Aggregate in a Gold model instead.
- Stagger heavy jobs and run large ingestion outside source business hours.
Where does the code live, and how do you avoid losing work?
The Git repository (dbt-models) is the source of truth; the DBT Editor saves straight to it. The terminal workspace is a temporary clone that is wiped when the dbt shell restarts (upgrades, maintenance, node moves, or 30 minutes idle on installs with scale-down). Committed but unpushed work is lost, so push after every change:
git add -A && git commit -m "silver: dedupe customers" && git push
git status -sb # must not show [ahead N]
How do you schedule dbt?
Create a job in Data Engineering → Jobs. Transform only runs snapshots and Bronze, Silver or Gold models, selected by layer, folder or model. Ingest + Transform runs a Connect pipeline, then dbt, then an optional cube refresh. Each dbt node runs on Spark by default or on Trino. Four rules keep scheduled projects healthy:
- Commit packages. Air-gapped installs cannot run
dbt depsat schedule time:git add -f packages.yml package-lock.yml dbt_packages/. - Keep the whole project parsing. Scheduling parses every model, so one broken model breaks every job.
- Stay compatible with the scheduler's dbt version, which can be older than the terminal's.
- Re-save the job after changing models so its project copy refreshes. A run that "succeeds" in milliseconds with no tasks means the copy is stale.
How do lineage and run history work?
UI and terminal runs both appear under Atlas → Runs, and each run writes metadata back to Atlas, so lineage traces a Gold metric back to its Bronze source. Lineage reads the target/ artifacts that runs commit, so do not add target/ to .gitignore. Where the catalog is installed, dbt description: fields appear in it.
Which errors come up most?
| Symptom | Cause | Fix |
|---|---|---|
Cannot set database in spark! | Trino-shaped model run on Spark | Use --target trino or rewrite the model |
INVALID_TABLE_PROPERTY | Spark-only config run on Trino | Switch the node's engine to Spark |
'dbt_expectations' is undefined | Packages not committed | Commit dbt_packages/ |
| Models vanished after restart | Committed but not pushed | Always push |
[ahead N] after UI runs | Automatic artifact push failed | Run git push |
Warehouse 1045 Access denied | Credential rotated mid-session | Open a new DBT Editor session |
Frequently asked questions
Which dbt adapters does DataNature use?
dbt-spark for the Spark 4 Connect target, dbt-trino for Trino, and a warehouse adapter for DN Warehouse. Credentials are injected and refreshed by the platform.
Can I bring an existing dbt project from Snowflake or Databricks?
Yes. Move the project into the Git repository, switch the target, and rewrite engine-specific functions and configurations. ANSI SQL models usually carry over unchanged.
Does dbt write Iceberg tables?
Yes. Spark and Trino targets write Iceberg tables in the lakehouse catalog.
Sources
- DataNature documentation: transformation with dbt
- DataNature documentation: jobs overview
- DataNature documentation: semantic layer
How does DataNature Atlas keep this secure and compliant?
Governance is part of the architecture, not a separate tool bolted on later. DataNature Atlas is the platform's governance module:
- Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
- Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
- Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
- Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
- Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.
These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.
Platform components in this article
Learn more about each DataNature product and module mentioned here.
Products
See it on your own data.
Book a demoDemos and quotes are handled by Dlytica, the company behind DataNature.