AI and agents
Private RAG on governed lakehouse data: how DocuContext, Milvus, cubes and Voir fit together
Private retrieval-augmented generation keeps every step inside your boundary: documents are parsed and embedded by DocuContext into a Milvus vector store, structured questions are answered from governed cubes over Gold tables, and Voir agents combine both under scoped access. The model is called through an AI gateway pointed at your approved endpoint, and nothing is sent to an external embedding or LLM API.
Published . Last updated
On this page
Retrieval-augmented generation (RAG) is how most enterprise AI answers questions about company data: retrieve the relevant facts, then ask a model to answer from them. "Private RAG" means no step of that pipeline leaves your boundary. This guide shows how to build it on governed lakehouse data in DataNature, and why numbers and documents should be retrieved differently.
What is private RAG?
Private RAG runs ingestion, embedding, vector storage, retrieval and model inference on infrastructure you control. No document, chunk, embedding, prompt or answer passes through a third-party API. It is the default architecture for regulated teams that want generative AI over contracts, policies, call recordings and customer data.
Why should numbers and documents be retrieved differently?
Because vector search is good at finding relevant passages and bad at arithmetic. Asking a model to compute "revenue by branch last quarter" from retrieved text chunks invites wrong joins and invented totals. The reliable pattern is:
- Documents and audio → embeddings in a vector store → semantic retrieval of passages.
- Metrics and records → a semantic layer over Gold tables → governed SQL with defined measures.
An agent decides which path a question needs, and cites the source either way. In DataNature, Voir answers are most accurate when twins use cubes, because metrics are defined once and the agent does not have to infer joins.
What does the architecture look like?
| Step | Component | What happens |
|---|---|---|
| 1. Collect | DocuContext connectors | Files arrive from SharePoint, SFTP or a document management system, or are uploaded to a partition |
| 2. Parse | DocuContext | Text parsing, OCR for scans and speech recognition for audio |
| 3. Extract and validate | DocuContext | Fields are classified and extracted with a confidence score and a link to source, then checked against business rules |
| 4. Embed | DocuContext | Chunks are embedded and written to a governed vector store |
| 5. Store | Milvus | Vectors live in the in-cluster Milvus database |
| 6. Model metrics | Atlas semantic models | YAML cubes define measures and dimensions over Gold Iceberg tables |
| 7. Orchestrate | Voir Agent Meta-Harness | Each turn fetches tools, applies guards, retrieves knowledge and streams the answer |
| 8. Generate | AI gateway | Calls your configured model endpoint |
| 9. Govern | Atlas and Voir guardrails | Scope limits, PII redaction, audit of each answer |
Structured fields extracted in step 3 are also written back into the lakehouse, so a contract's renewal date or a claim amount becomes queryable data, not just retrievable text.
How do you build it, step by step?
- Create DocuContext partitions that mirror your access model, for example one per business unit or sensitivity level. Members are managed under Data & AI Security → DocuContext users.
- Load documents through a connector or upload. Check extraction confidence on a sample before scaling up.
- Model the numbers. Write cubes over Gold tables in
semantic-models/cubes/with measures such asSUM(net_amount)and dimensions such aschannelandorder_date. Build cubes on Gold, never on Bronze. - Configure the AI gateway to your approved model endpoint. Until an admin does this it reads "Not configured".
- Create a Voir twin for each audience. Add integrations (lakehouse, semantic layer, documents), set the data scope to "Selected only" and pick the catalog, schema and tables, then add skills with worked examples of good answers.
- Turn on PII guardrails under Data & AI Security so sensitive values are detected and redacted in answers.
- Test with real questions from each audience, including ones they should not be able to answer, and confirm the twin refuses.
How is access controlled in private RAG?
Three rules keep retrieval inside each user's entitlement:
- Scope twins narrowly. The Voir lakehouse connector queries through a read-only service account, not the chatting user, so a twin's table scope is its security boundary. Give each audience its own twin.
- Partition documents. Document-partition limits are enforced inside the tool call, before Milvus is queried, so a twin cannot retrieve from a partition it is not scoped to.
- Use cube security for numbers. Row predicates per role (for example
branch_code = 'KTM') are injected into every cube query.
Governance is enforced in code, not by asking the model to behave. That distinction matters when someone tries a prompt injection.
What does it cost to run privately?
The main cost driver is model inference, not retrieval. A single-node Milvus serves millions of vectors on modest hardware; the GPU budget depends on the model size and concurrency you need. Start with a mid-sized open-weight model for retrieval-grounded answers, which needs less reasoning capacity than open-ended chat, and measure answer quality on your own question set before scaling up.
Frequently asked questions
Which vector database does DataNature use?
Milvus, running inside the cluster. DocuContext writes embeddings to it, and Voir twins retrieve from it.
Can RAG answers cite their sources?
Yes. DocuContext keeps a link from every extracted field and chunk to its source document, and Voir returns source-grounded answers.
Can I use Cohere or another provider instead of a self-hosted model?
Yes. The AI gateway points at whichever endpoint you approve. Many teams route sensitive twins to an in-country or self-hosted model and use a provider only for non-sensitive work.
Sources
- DataNature documentation: BI, Voir AI and ML tools
- DataNature documentation: semantic layer
- DataNature documentation: data engineering (DocuContext ingestion)
- DataNature documentation: Agent Meta-Harness
How does DataNature Atlas keep this secure and compliant?
Agents are governed like users: scoped, masked, logged and unable to act without approval. DataNature Atlas is the platform's governance module:
- Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
- Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
- Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
- Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
- Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.
These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.
Platform components in this article
Learn more about each DataNature product and module mentioned here.
Products
See it on your own data.
Book a demoDemos and quotes are handled by Dlytica, the company behind DataNature.