# DataNature architecture, technically: the open engines behind a sovereign AI lakehouse

> DataNature is a Kubernetes-native lakehouse built on open engines. Data lands in Apache Iceberg tables on object storage, Spark 4 and dbt refine it from Bronze to Gold, Trino and DN Warehouse serve SQL, StreamHouse runs the real-time hot path, and Atlas applies access policies, masking, lineage and audit across every engine. AI runs on the same governed data through experiment tracking, a model registry, a vector store and Voir agents.

Source: https://datanature.ai/blog/datanature-architecture-technical-deep-dive
Publisher: Dlytica Inc. (DataNature is a Dlytica company)
Updated: 2026-11-05

DataNature is a lakehouse you can run inside your own jurisdiction, on your own infrastructure if you need to. Under the product names sits a stack of open engines that a platform team can inspect, operate and leave. This article walks through that stack layer by layer, from ingestion to agents, and explains the design choices behind it.

## What is the DataNature architecture in one paragraph?

DataNature is one governed flow in four stages: **ingest**, **store and engineer**, **decide**, and **consume and activate**. Connect, StreamHouse and DocuContext land data in a Bronze layer of Apache Iceberg tables. A dbt and Spark workbench refines it to Silver and Gold. Trino answers interactive SQL and DN Warehouse serves low-latency dashboards and metrics, while StreamHouse handles the real-time hot path. DataNature AI and AI360 Studio build models and decisions, and Voir agents, BI dashboards and third-party apps consume the result. [Atlas](/platform/atlas) wraps every stage with a catalog, access policies, masking, lineage and audit.

## Which open engines does each layer use?

| Layer | DataNature component | Open technology underneath |
|---|---|---|
| Runtime | DataNature Platform | Kubernetes, containerised services, multi-tenant isolation |
| Storage | DataNature Lake | S3-compatible object storage (managed, or your S3, Azure Blob, Google Cloud Storage or on-premises store) |
| Table format | Lakehouse | Apache Iceberg by default; Delta Lake and Apache Hudi optional |
| Batch ingestion | Connect | JDBC extraction into Iceberg with watermark-based incremental loads |
| Real-time hot path | StreamHouse | Managed Apache Kafka, CDC capture, schema registry, exactly-once writes to Iceberg |
| Documents and audio | DocuContext | OCR, speech recognition, classification and extraction, embeddings |
| Transformation | Data Engineering | dbt (targets for Spark, Trino and the warehouse), Apache Spark 4 via Spark Connect, Python |
| Interactive SQL | Lakehouse query engine | Trino (MPP, federation across sources) |
| Serving warehouse | DN Warehouse | Low-latency OLAP engine with materialized views |
| Operational database | Lakebase | Managed PostgreSQL |
| Semantic layer | Atlas semantic models | YAML cubes compiled to warehouse SQL, served over REST and MCP |
| BI | DataNature BI | Apache Superset, plus Power BI or Tableau |
| Machine learning | DataNature AI | Experiment tracking, model registry, evaluation and tracing; Milvus vector database; Redis Stack cache |
| Agents | Voir | Agent Meta-Harness, connectors, MCP servers, guardrails |
| Catalog and lineage | Atlas | DataHub-based catalog where installed, dbt artifacts, run history |
| Identity and secrets | Platform security | Keycloak (OIDC, SAML 2.0), HashiCorp Vault |
| DevOps | DataNature DevOps | Git, Terraform, CI/CD pipelines, monitoring |

Every engine on that list has a public specification or an open-source code base. That is the practical meaning of "no lock-in": your Iceberg tables stay readable by any Iceberg engine if you leave.

## How does data move from source to Gold?

Batch data from Connect and real-time events from StreamHouse both land in Bronze. Data then follows the medallion pattern, and promotion between layers is explicit and validated.

1. **Bronze** is a raw, immutable copy of the source. Connect writes each source table to `lakehouse.bronze_<destination>.<table>` as an Iceberg table. Columns are nullable on purpose, so a source quirk never blocks a load.
2. **Silver** is cleansed and conformed. dbt models de-duplicate, cast types, handle nulls and enforce keys with `not_null`, `unique` and `relationships` tests.
3. **Gold** is business-ready facts and dimensions, certified and version-controlled. Dashboards, cubes, models and agents read Gold, never Bronze.

Lower layers are never deleted, so you can replay any transformation from source fidelity. That matters for audits and for migrations, where you reconcile old and new numbers.

## Which compute types does DataNature use, and why?

DataNature separates compute by the job it does, and every compute type works on the same Iceberg tables, so no data is copied between systems.

| Compute type | Engine | Best for | Typical latency |
|---|---|---|---|
| Real-time hot path | StreamHouse (managed Kafka, CDC, in-flight processing) | Fraud scoring, alerts, live operational views | Seconds |
| Batch and transformation | Apache Spark 4 via Spark Connect, dbt | Bronze to Gold pipelines, feature engineering, large backfills | Minutes to hours |
| Interactive SQL | Trino (MPP, federated) | Ad hoc analysis, exploration, JDBC clients, joins across sources | Seconds |
| Serving and metrics | DN Warehouse (low-latency OLAP with materialized views) | Dashboards, semantic-layer cubes, agent answers | Sub-second to seconds |
| Operational | Lakebase (managed PostgreSQL) | Applications and agents that read and write state | Milliseconds |

- **Iceberg** underneath gives ACID transactions, schema and partition evolution without rewriting files, and time travel through snapshots. Time travel makes regulator reporting reproducible: rerun last quarter's report against last quarter's snapshot.
- **StreamHouse** is the hot path. It captures inserts, updates and deletes from source logs, streams events through managed Kafka with a schema registry, and writes to Iceberg with exactly-once, ordered, de-duplicated delivery, so real-time and historical data meet in the same tables.
- **Spark 4** does the heavy lifting. dbt jobs default to the Spark target for new projects.
- **Trino** federates: one query can join Iceberg tables with a Postgres or warehouse table, which helps during migrations.
- **DN Warehouse** serves dashboards and cube queries. Cubes can be materialized as warehouse views, and matching queries are rewritten to them automatically.

The alternative, one engine for everything, forces a trade-off between batch throughput, streaming freshness and dashboard latency. Separating compute types over one table format avoids it.

## How is governance enforced across different engines?

Atlas applies one policy model and enforces it at the engines that serve users. In the Data Governance console an admin picks the engine (Lakehouse on Trino, or DN Warehouse) and defines:

- **Access grants** on catalog, schema, table or column, to users or groups
- **Row filters**, a SQL predicate per user or group, such as `branch_code = 'KTM'`
- **Column masks**: null, redact, show first or last four, hash (consistent, so still joinable), year only, or exempt
- **Cube security**, a row predicate per role injected into every semantic-layer query
- **Audit** of who accessed what

Policies take effect in about 30 to 60 seconds. They are enforced in the SQL workbench, CSV export, external JDBC clients through the query endpoint, semantic-layer cubes and the pre-configured Superset connections, which impersonate the logged-in user. Surfaces that run under a shared identity, such as notebooks, Spark and dbt, are controlled by page access instead. The design rule is simple: give people who must only see filtered data access to the enforced surfaces only.

## How are identity, secrets and encryption handled?

- **Identity**: single sign-on with SAML 2.0, OpenID Connect or Google, multi-factor authentication, and enterprise directories such as Active Directory or Okta. Keycloak provides OIDC.
- **Secrets**: HashiCorp Vault holds credentials in the KV v2 engine with versioning, issues short-lived dynamic database credentials that rotate automatically, and scopes secret paths and policies per tenant.
- **Encryption**: AES-256 at rest by default, TLS 1.2 or higher in transit, and per-tenant data keys through Vault's Transit engine.
- **Isolation**: workspaces are always isolated from each other, with dedicated compute and per-workspace storage.

Because the control plane runs inside your deployment, these components run inside your jurisdiction too. Query history, lineage, secrets and audit logs are not shipped to a vendor's region.

## How does AI fit on the same data?

AI reads the governed Gold layer and the governed document store instead of building its own copies.

- **DataNature AI** provides a feature store built from Gold data, model training and MLOps. Experiment tracking and a versioned model registry record every training run; Milvus stores embeddings for semantic search and retrieval-augmented generation (RAG).
- **DocuContext** turns documents and call audio into structured fields and embeddings through classify, extract, validate and integrate steps, each field carrying a confidence score and a link to source.
- **AI360 Studio** turns models into decisions: segmentation, decisioning, a semantic layer, reverse ETL back to operational systems, and a data access gateway.
- **Voir** runs role-based agents. Its Agent Meta-Harness rebuilds each agent on every turn: it fetches tools from live connectors, wraps them in governance guards, composes the prompt, retrieves knowledge and streams the answer. Document-partition and SQL-allowlist limits are enforced inside the tool call, before any downstream system is reached.

Models are called through an AI gateway that an admin points at your approved model endpoint, which can be a self-hosted model, a Dlytica-managed model or a provider such as Cohere.

## Where can DataNature run?

The same containerised platform runs in six deployment models: managed, dedicated, hosted in a sovereign partner cloud, in your own cloud account (AWS, Azure, Google Cloud, OVHcloud, IBM Cloud), on-premises, or fully air-gapped. Terraform and CI/CD templates make environments reproducible, and dev, test and production are kept separate.

## What are the current limits to plan around?

Being specific about limits saves design time:

- Connect supports four source databases today: Oracle, SQL Server, MySQL and PostgreSQL. Other connectors are in preview.
- Interactive SQL returns up to 10,000 rows; CSV export is capped at 1 GiB.
- Semantic queries return 100 rows by default and 1,000 at most.
- Each workspace has one shared Spark server with a platform-fixed executor size.
- The model registry, Milvus and Redis Stack run as single in-cluster instances.

## Frequently asked questions

### Is DataNature built on open source?

Yes. The storage format (Iceberg), engines (Spark, Trino, Kafka, PostgreSQL), transformation (dbt), BI (Superset), ML tooling (Milvus and open MLOps components) and security components (Keycloak, Vault) are open source. DataNature adds the integration, governance, operations and support.

### Does the control plane run in my environment?

Yes. The control plane runs inside your deployment, so metadata, lineage, query history, secrets and audit logs stay in the same jurisdiction as the data.

### Can I query DataNature tables from another engine?

Yes. Tables are Apache Iceberg, so any Iceberg-compatible engine with the right credentials can read them. External SQL clients can also connect over JDBC through the Trino endpoint, with the same row filters and masks applied.

## Sources

- [DataNature documentation: architecture](https://docs.datanature.ai/docs/developers/platform/architecture)
- [DataNature documentation: data warehouse](https://docs.datanature.ai/docs/developers/data-management/data-warehouse)
- [DataNature documentation: access and policies](https://docs.datanature.ai/docs/developers/governance/access-and-policies)
- [DataNature documentation: security overview](https://docs.datanature.ai/docs/security/overview)
- [Apache Iceberg](https://iceberg.apache.org/)
