Lakehouse architecture
Migrating regulated workloads from Databricks for data sovereignty
Migrating regulated workloads from Databricks to DataNature for sovereignty usually means moving Spark, SQL and dbt code with modest changes, keeping Delta tables as they are, re-creating jobs and permissions, and reconciling outputs with lineage before cutover. Many teams keep Databricks for public-cloud work and run both.
Published . Last updated
On this page
What moves easily?
Spark code, SQL, dbt projects and Delta tables, which DataNature supports natively.
What needs re-creating?
Jobs and workflows, Unity Catalog permissions, dashboards and any Databricks-only functions.
How do you reduce risk?
Run old and new in parallel and reconcile outputs with Atlas lineage before switching consumers, workload by workload.
Do we have to leave Databricks entirely?
No. Many organizations keep Databricks for public-cloud analytics and use DataNature for data that must stay in-country.
What maps to what, technically?
| On Databricks | On DataNature | Migration note |
|---|---|---|
| Delta tables | Iceberg tables (Delta optional) | Read Delta during transition; rewrite to Iceberg by domain |
| Lakeflow Connect, Auto Loader | Connect, StreamHouse | Connect for Oracle, SQL Server, MySQL, PostgreSQL; StreamHouse for CDC and events |
| Notebooks (PySpark, Spark SQL) | Spark 4 via Spark Connect in the workbench | Review Databricks-only APIs such as dbutils |
| dbt on Databricks | dbt with --target spark or --target trino | Change the adapter target; replace Spark-only configs on Trino models |
| Jobs and workflows | DataNature Jobs | Ingest, dbt and cube-refresh nodes; one pipeline or model per job |
| Unity Catalog grants, row filters, masks | Atlas grants, row filters, mask types | Scope policies to schemas; test with non-admin accounts |
| Databricks SQL dashboards | DataNature BI (Superset), Power BI, Tableau | Keep "Impersonate logged-in user" on for per-user rules |
| MLflow | DataNature AI experiment tracking and registry | Export and re-register models and experiments |
| Vector Search | Milvus | Re-embed documents through DocuContext |
Run old and new side by side, reconcile Gold metrics per period, then cut over one domain at a time. The immutable Bronze layer lets you reprocess safely if a mapping was wrong. For the deployment background, see can Databricks run on-premise?.
How does DataNature Atlas keep this secure and compliant?
Governance is part of the architecture, not a separate tool bolted on later. DataNature Atlas is the platform's governance module:
- Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
- Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
- Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
- Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
- Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.
These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.
Platform components in this article
Learn more about each DataNature product and module mentioned here.
Products
Platform modules
- AtlasGovernance for data and AI in one console
- Data Engineeringdbt and Spark, Bronze to Gold
- ComputeSpark and SQL engines, sized for the job
- ConnectEvery source, one governed landing zone
- StreamHouseReal-time data on the same tables
- BIDashboards on governed data
- LakehouseOpen tables in storage you control
See it on your own data.
Book a demoDemos and quotes are handled by Dlytica, the company behind DataNature.