AI and agents

MLOps and LLMOps on a sovereign platform: experiment tracking, model registry, tracing, evaluation and prompt versioning

DataNature AI covers the full model lifecycle inside your jurisdiction. Experiments log parameters, metrics and artifacts; a model registry versions every model with aliases such as champion and challenger and links it to its training data snapshot; tracing records every step of an LLM or agent call; evaluation scores answers with automated judges and human feedback; and prompts are versioned like code.

Published . Last updated

On this page

Models in regulated industries are judged by more than accuracy. Supervisors and model risk teams ask which data trained a model, which version made a decision, how it was validated, and how it is monitored. For generative AI they add new questions: which prompt was used, what the agent retrieved, and how answer quality is measured. This guide explains how DataNature AI answers both sets of questions.

What does the model lifecycle look like in DataNature AI?

StageCapabilityEvidence it produces
PrepareFeatures from certified Gold tables and the feature storeFeature definitions with lineage to source
ExperimentExperiment tracking of parameters, metrics, code version and artifactsA comparable record of every training run
RegisterModel registry with versions, aliases (champion, challenger) and descriptionsWhich version is in production and why
ValidateEvaluation runs against held-out data and business metricsValidation results attached to the version
DeployReal-time or batch serving from a registry aliasEvery prediction traceable to a version
MonitorPredictions and outcomes written back to Iceberg; drift and performance trackedOngoing monitoring evidence
RetrainReproduce training on a new Iceberg snapshotA new version linked to its predecessor

How does experiment tracking work?

Each training run records its parameters, metrics, code version, environment and output artifacts, and the Iceberg snapshot of the data it used. Runs are grouped into experiments and can be compared side by side. Because the data snapshot is recorded, any run can be reproduced exactly, which is what model validators need.

How does the model registry support model risk management?

The registry holds every model version with:

  • Aliases such as champion and challenger, so serving points at an alias and promotion is a controlled change, not a redeploy.
  • Lineage to the training run, data snapshot and feature definitions.
  • Descriptions and tags for owner, use, risk tier and validation status.
  • History of every promotion and demotion with who approved it.

Together they form the model inventory regulators expect, kept inside your jurisdiction.

What is LLM and agent tracing?

Tracing records every step of a generative AI request as a tree of spans: the user question, retrieval queries and the passages returned, cube queries, tool calls, the prompt sent to the model, the model's response, guardrail decisions, and the final answer, each with latency and token counts. For agents, a trace shows exactly why an answer came out the way it did, which makes debugging, auditing and cost analysis practical.

How is answer quality evaluated?

Generative answers cannot be scored by accuracy alone. DataNature AI evaluates them with:

  • Evaluation datasets of real questions with reference answers or expected sources, built with business users.
  • Automated judges that score groundedness (is every claim supported by the retrieved sources?), relevance, correctness against the reference, and safety.
  • Custom scorers for domain rules, such as "never quote an interest rate without a source".
  • Human feedback captured from users' ratings and reviewer corrections, attached to the trace.
  • Regression runs whenever a prompt, model or retrieval setting changes, so quality changes are visible before release.

Why version prompts?

Because a prompt change can alter every answer an agent gives. Prompts are stored in a registry with versions, authors and change notes, linked to the evaluation results for each version. Agents reference a prompt version, so rollback is instant and every answer can be traced to the exact prompt that produced it.

How does this fit with the rest of the platform?

  • Data: features and evaluation datasets come from governed Gold tables.
  • Gateway: the AI gateway routes model calls and logs tokens and cost per call.
  • Agents: Voir twins reference registered prompts and models, and every turn produces a trace.
  • Governance: DataNature Atlas records lineage from data to model to decision and controls who can register, promote or serve models.

Frequently asked questions

Can we bring models trained elsewhere?

Yes. Register them with their metadata and validation evidence, then serve them like any other version.

Does tracing store sensitive data?

Traces are stored inside your deployment and governed like other data, with PII guardrails and access policies applied.

How does this help with OSFI E-23 or SR 11-7?

It provides the inventory, lineage, validation records, monitoring and change history those frameworks ask for, generated as a by-product of normal work.

Sources

How does DataNature Atlas keep this secure and compliant?

Agents are governed like users: scoped, masked, logged and unable to act without approval. DataNature Atlas is the platform's governance module:

  • Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
  • Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
  • Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
  • Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
  • Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.

These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.

See it on your own data.

Book a demo

Demos and quotes are handled by Dlytica, the company behind DataNature.