# dbt on Spark and Trino with Apache Iceberg: building medallion pipelines that stay fast and correct

> DataNature runs dbt against three targets: Spark 4 Connect for heavy transformations into Iceberg, Trino for lighter SQL and federated sources, and DN Warehouse for low-latency serving tables. Projects live in Git, follow a staging, silver, intermediate and marts layout, de-duplicate in Silver, use incremental models for large facts, and commit packages so scheduled jobs run offline.

Source: https://datanature.ai/blog/dbt-spark-trino-iceberg-medallion-pipelines
Publisher: Dlytica Inc. (DataNature is a Dlytica company)
Updated: 2026-11-05

dbt is the transformation layer most data teams already know. On a lakehouse it gains a choice it does not have on a single warehouse: which engine runs each model. This guide covers how dbt works in DataNature, how to pick a target per model, and the habits that keep scheduled pipelines fast and correct.

## Which engines can a dbt model run on?

| Target | Engine | Writes to | Use for |
|---|---|---|---|
| `spark` | Spark 4 Connect | `lakehouse.<schema>` (Iceberg) | Heavy transforms; the default for new projects and jobs |
| `trino` | Trino | `lakehouse.<schema>` (Iceberg) | Lighter SQL, and models that read federated sources such as Postgres or the warehouse |
| `dnw` | DN Warehouse (serving compute) | Warehouse database | Serving-layer tables in `models/warehouse/` |

Use one engine per model. Spark-only configuration such as `file_format='iceberg'` fails on Trino with `INVALID_TABLE_PROPERTY`, and models that read relational catalogs fail on Spark with `Cannot set database in spark!`.

Always pass the target in the terminal. A bare `dbt run` uses the `profiles.yml` default, which is Trino:

```bash
dbt run --target spark --select silver.orders_clean+
dbt build --target trino --select tag:finance
```

## How should the project be laid out?

```text
models/
  staging/        # thin views over lakehouse.bronze_<src>.*: rename, cast
  silver/         # cleaned, de-duplicated, conformed entities
  intermediate/   # reusable joins
  marts/          # gold: facts and dimensions per domain (core/, finance/, ...)
  warehouse/      # dnw-only models (build with --target dnw)
snapshots/        # SCD2 history
tests/  macros/  seeds/
```

With the default profile schema `default`, `+schema: silver` builds into `lakehouse.default_silver` and `+schema: gold` into `lakehouse.default_gold`. Declare Bronze tables as sources so lineage and freshness work:

```yaml
sources:
  - name: bronze_crm
    database: lakehouse
    schema: bronze_crm
    tables:
      - name: customers
        loaded_at_field: last_modified
```

## Where should de-duplication and quality checks happen?

In Silver. Bronze is nullable and append-friendly by design, so Silver is where the data becomes trustworthy. Keep the latest row per key:

```sql
select * from (
  select *,
         row_number() over (partition by id order by last_modified desc) as rn
  from {{ source('bronze_crm', 'customers') }}
) where rn = 1
```

Then enforce it with tests: `not_null` and `unique` on Silver keys, and `relationships` between facts and dimensions in Gold.

## How do you keep large models fast?

- **Use incremental models for large facts**: `materialized='incremental'`, a `unique_key`, and a filter on the load timestamp. Full rebuilds of big tables are the most common cause of slow jobs.
- **Pre-aggregate in Gold** whatever dashboards and cubes need. Cubes sit on Gold, never on Bronze.
- **Avoid one-to-many joins in cubes**; they fan out and inflate sums. Aggregate in a Gold model instead.
- **Stagger heavy jobs** and run large ingestion outside source business hours.

## Where does the code live, and how do you avoid losing work?

The Git repository (`dbt-models`) is the source of truth; the DBT Editor saves straight to it. The terminal workspace is a temporary clone that is wiped when the dbt shell restarts (upgrades, maintenance, node moves, or 30 minutes idle on installs with scale-down). Committed but unpushed work is lost, so push after every change:

```bash
git add -A && git commit -m "silver: dedupe customers" && git push
git status -sb   # must not show [ahead N]
```

## How do you schedule dbt?

Create a job in Data Engineering → Jobs. **Transform only** runs snapshots and Bronze, Silver or Gold models, selected by layer, folder or model. **Ingest + Transform** runs a Connect pipeline, then dbt, then an optional cube refresh. Each dbt node runs on Spark by default or on Trino. Four rules keep scheduled projects healthy:

1. **Commit packages.** Air-gapped installs cannot run `dbt deps` at schedule time: `git add -f packages.yml package-lock.yml dbt_packages/`.
2. **Keep the whole project parsing.** Scheduling parses every model, so one broken model breaks every job.
3. **Stay compatible with the scheduler's dbt version**, which can be older than the terminal's.
4. **Re-save the job after changing models** so its project copy refreshes. A run that "succeeds" in milliseconds with no tasks means the copy is stale.

## How do lineage and run history work?

UI and terminal runs both appear under [Atlas](/platform/atlas) → Runs, and each run writes metadata back to Atlas, so lineage traces a Gold metric back to its Bronze source. Lineage reads the `target/` artifacts that runs commit, so do not add `target/` to `.gitignore`. Where the catalog is installed, dbt `description:` fields appear in it.

## Which errors come up most?

| Symptom | Cause | Fix |
|---|---|---|
| `Cannot set database in spark!` | Trino-shaped model run on Spark | Use `--target trino` or rewrite the model |
| `INVALID_TABLE_PROPERTY` | Spark-only config run on Trino | Switch the node's engine to Spark |
| `'dbt_expectations' is undefined` | Packages not committed | Commit `dbt_packages/` |
| Models vanished after restart | Committed but not pushed | Always push |
| `[ahead N]` after UI runs | Automatic artifact push failed | Run `git push` |
| Warehouse `1045 Access denied` | Credential rotated mid-session | Open a new DBT Editor session |

## Frequently asked questions

### Which dbt adapters does DataNature use?

dbt-spark for the Spark 4 Connect target, dbt-trino for Trino, and a warehouse adapter for DN Warehouse. Credentials are injected and refreshed by the platform.

### Can I bring an existing dbt project from Snowflake or Databricks?

Yes. Move the project into the Git repository, switch the target, and rewrite engine-specific functions and configurations. ANSI SQL models usually carry over unchanged.

### Does dbt write Iceberg tables?

Yes. Spark and Trino targets write Iceberg tables in the `lakehouse` catalog.

## Sources

- [DataNature documentation: transformation with dbt](https://docs.datanature.ai/docs/developers/data-management/data-engineering/transformation-dbt)
- [DataNature documentation: jobs overview](https://docs.datanature.ai/docs/developers/data-management/data-engineering/datanature-jobs/jobs-overview)
- [DataNature documentation: semantic layer](https://docs.datanature.ai/docs/developers/data-management/data-warehouse/semantic-layer)

## How does DataNature Atlas keep this secure and compliant?

Governance is part of the architecture, not a separate tool bolted on later. [DataNature Atlas](/platform/atlas) is the platform's governance module:

- **Knows your data**: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
- **Controls access**: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
- **Governs AI**: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
- **Proves it**: lineage from source to metric to decision, and an audit log of who or what accessed which data.
- **Stays local**: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.

These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read [DataNature Atlas in depth](/blog/datanature-atlas-governance-rbac-abac-pii-masking) or see [security and compliance](/security).
