AI and agents
MLOps and LLMOps on a sovereign platform: experiment tracking, model registry, tracing, evaluation and prompt versioning
DataNature AI covers the full model lifecycle inside your jurisdiction. Experiments log parameters, metrics and artifacts; a model registry versions every model with aliases such as champion and challenger and links it to its training data snapshot; tracing records every step of an LLM or agent call; evaluation scores answers with automated judges and human feedback; and prompts are versioned like code.
Published . Last updated
On this page
Models in regulated industries are judged by more than accuracy. Supervisors and model risk teams ask which data trained a model, which version made a decision, how it was validated, and how it is monitored. For generative AI they add new questions: which prompt was used, what the agent retrieved, and how answer quality is measured. This guide explains how DataNature AI answers both sets of questions.
What does the model lifecycle look like in DataNature AI?
| Stage | Capability | Evidence it produces |
|---|---|---|
| Prepare | Features from certified Gold tables and the feature store | Feature definitions with lineage to source |
| Experiment | Experiment tracking of parameters, metrics, code version and artifacts | A comparable record of every training run |
| Register | Model registry with versions, aliases (champion, challenger) and descriptions | Which version is in production and why |
| Validate | Evaluation runs against held-out data and business metrics | Validation results attached to the version |
| Deploy | Real-time or batch serving from a registry alias | Every prediction traceable to a version |
| Monitor | Predictions and outcomes written back to Iceberg; drift and performance tracked | Ongoing monitoring evidence |
| Retrain | Reproduce training on a new Iceberg snapshot | A new version linked to its predecessor |
How does experiment tracking work?
Each training run records its parameters, metrics, code version, environment and output artifacts, and the Iceberg snapshot of the data it used. Runs are grouped into experiments and can be compared side by side. Because the data snapshot is recorded, any run can be reproduced exactly, which is what model validators need.
How does the model registry support model risk management?
The registry holds every model version with:
- Aliases such as
championandchallenger, so serving points at an alias and promotion is a controlled change, not a redeploy. - Lineage to the training run, data snapshot and feature definitions.
- Descriptions and tags for owner, use, risk tier and validation status.
- History of every promotion and demotion with who approved it.
Together they form the model inventory regulators expect, kept inside your jurisdiction.
What is LLM and agent tracing?
Tracing records every step of a generative AI request as a tree of spans: the user question, retrieval queries and the passages returned, cube queries, tool calls, the prompt sent to the model, the model's response, guardrail decisions, and the final answer, each with latency and token counts. For agents, a trace shows exactly why an answer came out the way it did, which makes debugging, auditing and cost analysis practical.
How is answer quality evaluated?
Generative answers cannot be scored by accuracy alone. DataNature AI evaluates them with:
- Evaluation datasets of real questions with reference answers or expected sources, built with business users.
- Automated judges that score groundedness (is every claim supported by the retrieved sources?), relevance, correctness against the reference, and safety.
- Custom scorers for domain rules, such as "never quote an interest rate without a source".
- Human feedback captured from users' ratings and reviewer corrections, attached to the trace.
- Regression runs whenever a prompt, model or retrieval setting changes, so quality changes are visible before release.
Why version prompts?
Because a prompt change can alter every answer an agent gives. Prompts are stored in a registry with versions, authors and change notes, linked to the evaluation results for each version. Agents reference a prompt version, so rollback is instant and every answer can be traced to the exact prompt that produced it.
How does this fit with the rest of the platform?
- Data: features and evaluation datasets come from governed Gold tables.
- Gateway: the AI gateway routes model calls and logs tokens and cost per call.
- Agents: Voir twins reference registered prompts and models, and every turn produces a trace.
- Governance: DataNature Atlas records lineage from data to model to decision and controls who can register, promote or serve models.
Frequently asked questions
Can we bring models trained elsewhere?
Yes. Register them with their metadata and validation evidence, then serve them like any other version.
Does tracing store sensitive data?
Traces are stored inside your deployment and governed like other data, with PII guardrails and access policies applied.
How does this help with OSFI E-23 or SR 11-7?
It provides the inventory, lineage, validation records, monitoring and change history those frameworks ask for, generated as a by-product of normal work.
Sources
- DataNature documentation: machine learning
- DataNature: OSFI Guideline E-23 explained
- DataNature: AI gateway for LLM cost control and guardrails
How does DataNature Atlas keep this secure and compliant?
Agents are governed like users: scoped, masked, logged and unable to act without approval. DataNature Atlas is the platform's governance module:
- Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
- Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
- Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
- Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
- Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.
These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.
Platform components in this article
Learn more about each DataNature product and module mentioned here.
See it on your own data.
Book a demoDemos and quotes are handled by Dlytica, the company behind DataNature.