Lakehouse architecture

StreamHouse: the real-time hot path on a sovereign lakehouse, with architecture and use cases

StreamHouse is DataNature's real-time hot path. It captures inserts, updates and deletes from source database logs and event streams, runs them through managed Apache Kafka with a schema registry and in-flight processing, and writes them to Apache Iceberg with exactly-once, ordered, de-duplicated delivery. Fraud models, alerts and live dashboards act on data seconds old, in the same tables that hold history.

Published . Last updated

On this page

Batch pipelines answer "what happened yesterday". Some decisions cannot wait that long: a suspicious card transaction, a SIM-box pattern on a telecom network, a stock-out at a distributor, a machine drifting out of tolerance. StreamHouse is the part of DataNature built for those decisions. This article explains how it works, how it fits next to batch, and which use cases belong on the hot path.

What is a hot path, and why does a lakehouse need one?

A hot path is the low-latency route that data takes when a decision depends on events from the last few seconds. The cold path is the scheduled batch route for history, reconciliation and heavy transformation. Older architectures ran the two on separate systems and spent years reconciling them. StreamHouse writes the hot path into the same Apache Iceberg tables the batch path uses, so a fraud score computed in real time and the monthly loss report read the same facts.

How does StreamHouse work?

StageWhat happens
CaptureChange data capture reads row-level inserts, updates and deletes from source database logs; applications and devices publish events directly
TransportManaged Apache Kafka carries events in topics and partitions, preserving order per key
ContractsA schema registry holds the event contract, so producers cannot silently break consumers
Process in flightEvents are filtered, enriched and joined while moving, for example adding a customer's risk band to a transaction
WriteEvents land in Bronze Iceberg tables with exactly-once delivery, ordering and de-duplication
ActModels, rules, dashboards and Voir agents read the fresh data and trigger alerts or actions
GovernAtlas catalogs every stream and table, masks PII and records lineage, as for batch

Because StreamHouse tracks processing by checkpoint, it handles only events after the last committed position. Restarts resume where they stopped, without gaps or duplicates.

When should data go through StreamHouse instead of Connect?

RequirementConnect (batch)StreamHouse (hot path)
FreshnessMinutes to hoursSeconds
Captures every update and deleteOnly if the bookmark column changes on updateYes, from the source log
Best forScheduled loads, backfills, full refreshes, reconciliationOperational events, CDC replication, event-driven actions
Operational complexityLowerHigher for stateful work: joins, aggregations, late or out-of-order events
Typical sourcesOracle, SQL Server, MySQL, PostgreSQL tablesDatabase logs, Kafka topics, IoT telemetry, clickstream, payment events

Most programmes use both: StreamHouse for the handful of tables and events where seconds matter, Connect for everything else, and dbt to model both into the same Silver and Gold layers.

Which use cases belong on the hot path?

  • Card and payment fraud: score each authorization against recent velocity, device and merchant features, and hold or step up suspicious payments while the customer is still at the till.
  • Account takeover: detect a password reset, new device and payee change within minutes and freeze outbound transfers.
  • Telecom SIM-box and interconnect fraud: spot call patterns from network events as they happen and block numbers before losses build.
  • Network operations: correlate alarms and performance counters to raise one incident, not a hundred.
  • Distributor replenishment: watch orders and stock movements and alert before a retailer runs out.
  • Production yield and loss: flag sensor readings drifting out of range on a production line.
  • Customer service: show agents the customer's last few interactions and open tickets in real time.

How do real-time models get features?

The feature store is built from Gold data, and hot-path scoring needs the most recent values. A common pattern: StreamHouse lands raw events; in-flight processing maintains short-window aggregates (transactions in the last 10 minutes, distinct devices today); the model scores each event against those plus slower features from Gold; results and decisions are written back to Iceberg for audit and retraining. The model registry records which model version scored which event, so every decision is explainable after the fact.

How do you design a reliable hot path?

  1. Keep it narrow. Put only latency-critical tables and events on StreamHouse.
  2. Define contracts first. Register schemas before producers go live; evolve them compatibly.
  3. Key by the entity you decide on, such as card, account or subscriber, so per-entity order is preserved.
  4. Plan for late and out-of-order events with event-time windows and explicit lateness limits.
  5. Write decisions back. Store every score, rule hit and action next to the event for audit and model retraining.
  6. Reconcile with batch. Compare hot-path totals with the batch path daily; differences reveal dropped or duplicated events early.

Frequently asked questions

Is StreamHouse Kafka?

StreamHouse uses managed Apache Kafka for transport, plus change data capture, a schema registry, in-flight processing and exactly-once writes to Iceberg, all governed by Atlas and deployed inside your jurisdiction.

Does real-time data need a separate database?

No. StreamHouse writes to the same Iceberg tables as batch, and Lakebase (managed PostgreSQL) can serve operational state to applications and agents that need millisecond reads and writes.

Can StreamHouse run on-premises or air-gapped?

Yes. It is part of the DataNature platform and runs in every deployment model, including on-premises, telecom private clouds and air-gapped environments.

Sources

How does DataNature Atlas keep this secure and compliant?

Governance is part of the architecture, not a separate tool bolted on later. DataNature Atlas is the platform's governance module:

  • Knows your data: a catalog with automated PII detection and classification across Bronze, Silver and Gold.
  • Controls access: role-based grants plus attribute-based policies, row filters and column masks (null, redact, partial, hash, year only).
  • Governs AI: Voir agents are scoped principals; masks, row filters and PII guardrails apply to every tool call and answer.
  • Proves it: lineage from source to metric to decision, and an audit log of who or what accessed which data.
  • Stays local: Atlas runs inside your deployment, so policies, lineage and logs never leave your jurisdiction.

These controls support GDPR, PIPEDA, Quebec Law 25, CCPA, HIPAA and sector rules such as OSFI E-23. Read DataNature Atlas in depth or see security and compliance.

See it on your own data.

Book a demo

Demos and quotes are handled by Dlytica, the company behind DataNature.