Lakehouse architecture

dbt on Spark and Trino with Apache Iceberg: building medallion pipelines that stay fast and correct

DataNature runs dbt against three targets: Spark 4 Connect for heavy transformations into Iceberg, Trino for lighter SQL and federated sources, and DN Warehouse for low-latency serving tables. Projects live in Git, follow a staging, silver, intermediate and marts layout, de-duplicate in Silver, use incremental models for large facts, and commit packages so scheduled jobs run offline.

Published . Last updated

On this page

dbt is the transformation layer most data teams already know. On a lakehouse it gains a choice it does not have on a single warehouse: which engine runs each model. This guide covers how dbt works in DataNature, how to pick a target per model, and the habits that keep scheduled pipelines fast and correct.

Which engines can a dbt model run on?

TargetEngineWrites toUse for
sparkSpark 4 Connectlakehouse.<schema> (Iceberg)Heavy transforms; the default for new projects and jobs
trinoTrinolakehouse.<schema> (Iceberg)Lighter SQL, and models that read federated sources such as Postgres or the warehouse
dnwDN Warehouse (serving compute)Warehouse databaseServing-layer tables in models/warehouse/

Use one engine per model. Spark-only configuration such as file_format='iceberg' fails on Trino with INVALID_TABLE_PROPERTY, and models that read relational catalogs fail on Spark with Cannot set database in spark!.

Always pass the target in the terminal. A bare dbt run uses the profiles.yml default, which is Trino:

dbt run --target spark --select silver.orders_clean+
dbt build --target trino --select tag:finance

How should the project be laid out?

models/
  staging/        # thin views over lakehouse.bronze_<src>.*: rename, cast
  silver/         # cleaned, de-duplicated, conformed entities
  intermediate/   # reusable joins
  marts/          # gold: facts and dimensions per domain (core/, finance/, ...)
  warehouse/      # dnw-only models (build with --target dnw)
snapshots/        # SCD2 history
tests/  macros/  seeds/

With the default profile schema default, +schema: silver builds into lakehouse.default_silver and +schema: gold into lakehouse.default_gold. Declare Bronze tables as sources so lineage and freshness work:

sources:
  - name: bronze_crm
    database: lakehouse
    schema: bronze_crm
    tables:
      - name: customers
        loaded_at_field: last_modified

Where should de-duplication and quality checks happen?

In Silver. Bronze is nullable and append-friendly by design, so Silver is where the data becomes trustworthy. Keep the latest row per key:

select * from (
  select *,
         row_number() over (partition by id order by last_modified desc) as rn
  from {{ source('bronze_crm', 'customers') }}
) where rn = 1

Then enforce it with tests: not_null and unique on Silver keys, and relationships between facts and dimensions in Gold.

How do you keep large models fast?

  • Use incremental models for large facts: materialized='incremental', a unique_key, and a filter on the load timestamp. Full rebuilds of big tables are the most common cause of slow jobs.
  • Pre-aggregate in Gold whatever dashboards and cubes need. Cubes sit on Gold, never on Bronze.
  • Avoid one-to-many joins in cubes; they fan out and inflate sums. Aggregate in a Gold model instead.
  • Stagger heavy jobs and run large ingestion outside source business hours.

Where does the code live, and how do you avoid losing work?

The Git repository (dbt-models) is the source of truth; the DBT Editor saves straight to it. The terminal workspace is a temporary clone that is wiped when the dbt shell restarts (upgrades, maintenance, node moves, or 30 minutes idle on installs with scale-down). Committed but unpushed work is lost, so push after every change:

git add -A && git commit -m "silver: dedupe customers" && git push
git status -sb   # must not show [ahead N]

How do you schedule dbt?

Create a job in Data Engineering → Jobs. Transform only runs snapshots and Bronze, Silver or Gold models, selected by layer, folder or model. Ingest + Transform runs a Connect pipeline, then dbt, then an optional cube refresh. Each dbt node runs on Spark by default or on Trino. Four rules keep scheduled projects healthy:

  1. Commit packages. Air-gapped installs cannot run dbt deps at schedule time: git add -f packages.yml package-lock.yml dbt_packages/.
  2. Keep the whole project parsing. Scheduling parses every model, so one broken model breaks every job.
  3. Stay compatible with the scheduler's dbt version, which can be older than the terminal's.
  4. Re-save the job after changing models so its project copy refreshes. A run that "succeeds" in milliseconds with no tasks means the copy is stale.

How do lineage and run history work?

UI and terminal runs both appear under Atlas → Runs, and each run writes metadata back to Atlas, so lineage traces a Gold metric back to its Bronze source. Lineage reads the target/ artifacts that runs commit, so do not add target/ to .gitignore. Where the catalog is installed, dbt description: fields appear in it.

Which errors come up most?

SymptomCauseFix
Cannot set database in spark!Trino-shaped model run on SparkUse --target trino or rewrite the model
INVALID_TABLE_PROPERTYSpark-only config run on TrinoSwitch the node's engine to Spark
'dbt_expectations' is undefinedPackages not committedCommit dbt_packages/
Models vanished after restartCommitted but not pushedAlways push
[ahead N] after UI runsAutomatic artifact push failedRun git push
Warehouse 1045 Access deniedCredential rotated mid-sessionOpen a new DBT Editor session

Frequently asked questions

Which dbt adapters does DataNature use?

dbt-spark for the Spark 4 Connect target, dbt-trino for Trino, and a warehouse adapter for DN Warehouse. Credentials are injected and refreshed by the platform.

Can I bring an existing dbt project from Snowflake or Databricks?

Yes. Move the project into the Git repository, switch the target, and rewrite engine-specific functions and configurations. ANSI SQL models usually carry over unchanged.

Does dbt write Iceberg tables?

Yes. Spark and Trino targets write Iceberg tables in the lakehouse catalog.

Sources

How does DataNature Atlas keep this secure and compliant?

Governance is part of the architecture, not a separate tool bolted on later. DataNature Atlas is the platform's governance module:

  • Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
  • Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
  • Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
  • Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
  • Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.

These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.

See it on your own data.

Book a demo

Demos and quotes are handled by Dlytica, the company behind DataNature.