Sovereignty and compliance
Air-gapped lakehouse and private AI: a reference architecture that runs with no internet
An air-gapped lakehouse runs every dependency inside the enclave: object storage, Iceberg tables, Spark, Trino, the catalog, identity, secrets, a container registry mirror, model weights, the vector store and the LLM serving layer. Updates arrive on signed media. DataNature deploys this stack on Kubernetes with no outbound connection, including dbt packages committed to Git so scheduled jobs never call the internet.
Published . Last updated
On this page
Some workloads cannot touch the internet at all: central banks, defence suppliers, critical infrastructure, health records under strict local rules. For them, "private networking" is not enough; the platform has to work with no route out. This article lays out a reference architecture for an air-gapped lakehouse with private AI, and the operational details that decide whether it works on day 300, not just day one.
What does air-gapped actually mean?
An air-gapped system has no network path to the outside: no NAT gateway, no egress allowlist, no external DNS, no public certificate chain. Data and software cross the boundary only on removable media under a documented procedure. An "isolated" VPC with controlled egress is segregated, not air-gapped, and auditors treat the two differently.
Which components must live inside the enclave?
Everything the system calls at runtime.
| Layer | Inside the enclave | DataNature component |
|---|---|---|
| Compute runtime | Kubernetes cluster, internal container registry mirror | DataNature Platform |
| Storage | S3-compatible object store on local disks or appliances | DataNature Lake |
| Tables | Apache Iceberg tables and catalog metadata | Lakehouse |
| Processing | Spark 4, dbt with committed packages; Kafka for the real-time hot path | Data Engineering, StreamHouse |
| SQL and serving | Trino for interactive SQL, a low-latency serving warehouse | Lakehouse SQL, DN Warehouse |
| Governance | Catalog, policies, lineage, audit store | Atlas |
| Identity | OIDC/SAML provider tied to the internal directory | Keycloak |
| Secrets and keys | Secret store, transit encryption keys | HashiCorp Vault |
| Documents | OCR and speech models, extraction | DocuContext |
| Retrieval | Embedding model and vector database | Milvus |
| Models | Model weights and an inference server | AI gateway pointed at a local endpoint |
| Agents | Agent runtime, internal connectors only | Voir |
| Observability | Metrics, logs, alerting | DevOps monitoring |
| Certificates | Internal certificate authority | Platform ingress |
How does data get in and out?
Through defined transfer zones, never ad hoc.
- Inbound data from internal source systems flows over the internal network into Bronze through Connect (Oracle, SQL Server, MySQL and PostgreSQL today). External data arrives on media, is scanned in a staging zone, then loaded as files.
- Outbound results (regulatory returns, approved extracts) leave through an export zone with a review step. The CSV export limit in DataNature is 1 GiB per file, so plan large extracts as partitioned files.
- Metadata such as table schemas and lineage never needs to leave; Atlas keeps it inside.
How do you serve LLMs with no internet?
Stage everything a model needs before you need it.
- Model weights are imported on signed media into an internal model store, with a hash recorded at import.
- Tokenizers and embedding models are pulled from local paths, not from a model hub at start-up. Embedding SDKs that default to a cloud endpoint are a common silent leak; check network logs for unexpected DNS queries.
- An inference server (such as vLLM) runs on local GPUs. DataNature's AI gateway is configured to call that endpoint; until an admin configures it, the gateway shows "Not configured" and agents cannot call any model.
- Retrieval uses DocuContext to parse and embed documents into Milvus inside the cluster. Voir twins query the document partitions they are scoped to.
- Guardrails run locally: Voir enforces SQL allowlists and document partitions inside each tool call, and PII guardrails redact values in answers.
What breaks in air-gapped deployments, and how do you prevent it?
Most failures come from something quietly expecting the internet.
| Failure | Symptom | Prevention |
|---|---|---|
| Package managers at run time | dbt tests fail with 'dbt_expectations' is undefined | Commit packages.yml, package-lock.yml and dbt_packages/ to Git, because dbt deps cannot run at schedule time |
| Image pulls | Pods stuck in ImagePullBackOff after a node change | Mirror every image, pin digests, test node replacement |
| Hidden telemetry | DNS queries to external hosts | Block egress at the network layer and monitor DNS |
| Certificate expiry | Services fail on renewal day | Internal CA with tracked expiry dates |
| Long ingestion runs | Runs over 30 to 60 minutes fail with NotAuthorized as catalog sessions expire | Keep backfill chunks small, partition large tables by month |
| Update drift | Environments diverge over months | Treat each media drop as a release, applied through Terraform and CI/CD in test first |
How are updates delivered?
On a fixed cadence, monthly to quarterly, as signed bundles: container images, Helm charts or manifests, model weights and documentation. Each bundle is verified, applied to a test environment inside the enclave, then promoted. Because the platform is defined in Terraform and deployed through CI/CD, the same procedure rebuilds an environment from scratch, which is also your disaster-recovery drill.
What controls does this give auditors?
- Physical and network separation, with no outbound path
- Encryption at rest (AES-256) and in transit (TLS 1.2+), with keys in your Vault
- Row filters, column masks and access grants enforced in the SQL engines
- Lineage from source table to metric, and an audit log of who accessed what
- A model register with hashes, so you can prove which model produced which answer
Frequently asked questions
Can DataNature run fully air-gapped?
Yes. Air-gapped is one of the six DataNature deployment models. The whole stack, including the control plane, identity, secrets, models and vector store, runs inside the enclave.
Which LLMs can run air-gapped?
Any open-weight model your hardware can serve and your policy approves, served by a local inference server. The AI gateway calls whichever endpoint you configure.
How do analysts get BI in an air-gapped environment?
DataNature BI (Apache Superset) runs inside the cluster. Power BI Report Server or Tableau Server installed on-premises can also connect to the lakehouse or warehouse endpoints.
Sources
- TrueFoundry: air-gapped AI in highly regulated industries
- Alex Merced: building Apache Iceberg lakehouses that run without an internet connection
- DataNature documentation: transformation with dbt
- DataNature documentation: limits and known issues
How does DataNature Atlas keep this secure and compliant?
Every rule in this article is enforced in one place, inside your jurisdiction. DataNature Atlas is the platform's governance module:
- Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
- Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
- Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
- Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
- Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.
These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.
Platform components in this article
Learn more about each DataNature product and module mentioned here.
Products
Platform modules
- AtlasGovernance for data and AI in one console
- Data Engineeringdbt and Spark, Bronze to Gold
- ComputeSpark and SQL engines, sized for the job
- ConnectEvery source, one governed landing zone
- LakehouseOpen tables in storage you control
- BIDashboards on governed data
- StreamHouseReal-time data on the same tables
- DataNature AITrain, serve and monitor models in your jurisdiction
See it on your own data.
Book a demoDemos and quotes are handled by Dlytica, the company behind DataNature.