Lakehouse architecture

Can Databricks run on-premise? What is possible, what is not, and the sovereign alternative

No. Databricks has no supported on-premises or air-gapped deployment. The control plane is always operated by Databricks in its cloud, and even classic compute runs in your public cloud account, not your data centre. Teams that need the lakehouse inside their own walls run open engines themselves, or use a platform such as DataNature that deploys the full stack on-premises.

Published . Last updated

On this page

"Can we run Databricks on-premise?" comes up in almost every regulated evaluation. The short answer is no, and the reason is architectural rather than commercial. This guide explains how Databricks is split, what each deployment option actually keeps in your control, and how to get a Databricks-style lakehouse inside your own data centre.

Can Databricks be installed on-premises?

No. Databricks is delivered as a managed service on AWS, Azure and Google Cloud. There is no self-hosted, bare-metal or disconnected edition. The closest option, classic compute, runs clusters in your own cloud account, but that is still a public cloud, and the control plane remains with Databricks.

How is Databricks split between control plane and compute plane?

Databricks describes its architecture as a control plane plus a compute plane:

PlaneWhat it holdsWho operates itWhere it runs
Control planeWeb application, notebooks, job definitions, cluster management, workspace configurationDatabricksDatabricks' cloud account, in a region you pick
Classic compute planeClusters that process your dataDatabricks software in your cloud accountYour AWS, Azure or Google Cloud account
Serverless compute planeServerless SQL warehouses, serverless jobsDatabricksDatabricks' cloud account, regional

Your tables can sit in your own cloud storage, but the service that orchestrates them, stores workspace metadata and runs serverless features lives in an account you do not control. For many teams that is fine. For teams whose policy says metadata and processing must stay inside their own perimeter, it is the deciding fact.

What about Databricks AI features and data residency?

Databricks processes some AI features, which it calls Designated Services, on a "Geo" basis rather than a single region. Its documentation groups the United States, Brazil and Canada into one Americas Geo, and states that cross-Geo processing is enabled by default for workspaces in Geos outside the US and EU that do not have a compliance security profile. Admins can enforce in-Geo processing, but some assistive features then need it switched off to work. If your rule is "this country only", read those settings closely; a Geo is wider than a country.

Is private networking the same as on-premise?

No. Private Link, customer-managed VPCs and IP access lists reduce exposure, but traffic still flows to a vendor-operated control plane over a private route. That is network isolation, not an air gap. An air-gapped system has no route out at all, and every dependency (container images, models, packages, updates) is staged inside the boundary.

What are the real options if the platform must run on-premises?

There are three, and they differ in how much you assemble yourself.

  1. Build it yourself from open source. Apache Spark, Apache Iceberg or Delta Lake, an Iceberg REST catalog, Trino, Airflow, Superset and a vector database on your own Kubernetes. Full control, and a platform team to integrate, secure, upgrade and support every piece.
  2. Run an analytics engine without the platform. A single engine on your own hardware covers SQL analytics but not pipelines, governance, ML and agents together.
  3. Deploy a packaged sovereign lakehouse. A platform that ships those open engines pre-integrated, with governance and operations, and runs on-premises, in a private cloud or air-gapped. DataNature is one such platform.

How does DataNature map to Databricks concepts?

Databricks conceptDataNature equivalentUnderlying technology
Delta tablesLakehouse tablesApache Iceberg (Delta Lake optional)
Lakeflow ConnectConnectJDBC extraction into Iceberg, watermark incremental loads
Structured StreamingStreamHouseManaged Kafka, CDC, schema registry, exactly-once writes to Iceberg
Notebooks and SQL editorData Engineering workbenchSpark 4 Connect, dbt, Python, query editor
Databricks SQLInteractive SQL and serving computeTrino, DN Warehouse
Unity CatalogAtlasCatalog, row filters, column masks, lineage, audit
Lakeflow JobsDataNature JobsIngest, dbt and cube-refresh nodes on a schedule
LakebaseLakebaseManaged PostgreSQL
AI/BI dashboardsDataNature BIApache Superset, Power BI or Tableau
MLflow, Vector SearchDataNature AIExperiment tracking, model registry, Milvus
Genie, Agent BricksVoirAgent Meta-Harness, MCP connectors, guardrails

The difference is where it runs: every row of that table, control plane included, can be deployed in your data centre.

What does moving from Databricks to an on-premises lakehouse involve?

A typical path, in the order teams do it:

  1. Inventory notebooks, jobs, tables, permissions and dashboards. Classify which workloads must move and which can stay.
  2. Land data in Bronze. Source databases load through Connect; Delta tables can be read directly or rewritten to Iceberg.
  3. Port transformations. Spark SQL and PySpark carry over to Spark 4. dbt projects move with a target change (--target spark or --target trino).
  4. Recreate governance as Atlas grants, row filters and masks, scoped to schemas rather than whole catalogs.
  5. Reconcile. Compare row counts and key metrics per period between old and new Gold tables before cutover. The immutable Bronze layer lets you reprocess safely.
  6. Cut over by domain, not all at once, and keep the public-cloud platform for workloads that do not need to move.

Frequently asked questions

Does Databricks offer an air-gapped edition?

No. No supported Databricks deployment mode runs disconnected from Databricks' control plane.

Can I keep Databricks for some workloads and run DataNature for others?

Yes, and many organizations do. Both read open table formats, so approved data can move between them while regulated workloads stay in-country.

Will my Spark code run on DataNature?

Spark SQL and PySpark run on Apache Spark 4 in DataNature. Code that depends on Databricks-only runtime features needs review during migration.

Sources

How does DataNature Atlas keep this secure and compliant?

Governance is part of the architecture, not a separate tool bolted on later. DataNature Atlas is the platform's governance module:

  • Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
  • Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
  • Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
  • Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
  • Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.

These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.

See it on your own data.

Book a demo

Demos and quotes are handled by Dlytica, the company behind DataNature.