Lakehouse architecture

Migrating regulated workloads from Databricks for data sovereignty

Migrating regulated workloads from Databricks to DataNature for sovereignty usually means moving Spark, SQL and dbt code with modest changes, keeping Delta tables as they are, re-creating jobs and permissions, and reconciling outputs with lineage before cutover. Many teams keep Databricks for public-cloud work and run both.

Published . Last updated

On this page

What moves easily?

Spark code, SQL, dbt projects and Delta tables, which DataNature supports natively.

What needs re-creating?

Jobs and workflows, Unity Catalog permissions, dashboards and any Databricks-only functions.

How do you reduce risk?

Run old and new in parallel and reconcile outputs with Atlas lineage before switching consumers, workload by workload.

Do we have to leave Databricks entirely?

No. Many organizations keep Databricks for public-cloud analytics and use DataNature for data that must stay in-country.

What maps to what, technically?

On DatabricksOn DataNatureMigration note
Delta tablesIceberg tables (Delta optional)Read Delta during transition; rewrite to Iceberg by domain
Lakeflow Connect, Auto LoaderConnect, StreamHouseConnect for Oracle, SQL Server, MySQL, PostgreSQL; StreamHouse for CDC and events
Notebooks (PySpark, Spark SQL)Spark 4 via Spark Connect in the workbenchReview Databricks-only APIs such as dbutils
dbt on Databricksdbt with --target spark or --target trinoChange the adapter target; replace Spark-only configs on Trino models
Jobs and workflowsDataNature JobsIngest, dbt and cube-refresh nodes; one pipeline or model per job
Unity Catalog grants, row filters, masksAtlas grants, row filters, mask typesScope policies to schemas; test with non-admin accounts
Databricks SQL dashboardsDataNature BI (Superset), Power BI, TableauKeep "Impersonate logged-in user" on for per-user rules
MLflowDataNature AI experiment tracking and registryExport and re-register models and experiments
Vector SearchMilvusRe-embed documents through DocuContext

Run old and new side by side, reconcile Gold metrics per period, then cut over one domain at a time. The immutable Bronze layer lets you reprocess safely if a mapping was wrong. For the deployment background, see can Databricks run on-premise?.

How does DataNature Atlas keep this secure and compliant?

Governance is part of the architecture, not a separate tool bolted on later. DataNature Atlas is the platform's governance module:

  • Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
  • Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
  • Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
  • Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
  • Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.

These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.

See it on your own data.

Book a demo

Demos and quotes are handled by Dlytica, the company behind DataNature.