# Real-time CDC lakehouse with StreamHouse

> A real-time CDC lakehouse on DataNature captures inserts, updates and deletes from source databases and streams events through managed Kafka into Iceberg tables. StreamHouse provides a schema registry and transactional, idempotent writes, so fraud models, alerts and dashboards work on data that is seconds old.

Source: https://datanature.ai/solutions/real-time-cdc-lakehouse
Publisher: Dlytica Inc. (DataNature is a Dlytica company)
Updated: 2026-11-05

Database changes and events in your tables within seconds

## Components

- **Connect CDC**: Row-level change capture from Postgres, Oracle, SQL Server and MySQL logs
- **StreamHouse**: Managed Kafka, schema registry, in-flight processing, transactional writes
- **Lakehouse**: Real-time Bronze tables in Delta or Iceberg
- **Atlas**: Lineage and access control on streaming data

## Steps

1. **Capture changes**: CDC reads database logs and publishes each change as an event.
2. **Stream and validate**: Events pass through managed Kafka with schema contracts and in-flight processing.
3. **Write exactly once**: Kafka transactions and checkpointed, idempotent table commits give exactly-once writes into tables. Source capture is at-least-once; duplicates are removed by key on write.
4. **Serve live and history**: The same tables serve real-time scoring, SQL and model training.

## Documentation

- [Batch vs real-time](https://docs.datanature.ai/docs/developers/data-management/data-engineering)

## When should we use real-time instead of batch?

When latency matters: fraud, transaction monitoring, network events, IoT. Batch stays simpler for scheduled loads and reconciliation.

## What makes streaming hard, and how is it handled?

Joins, deduplication and late or out-of-order events. StreamHouse preserves ordering per key, removes duplicates on write, and checkpoints track what has been processed.

## Can streaming and batch share tables?

Yes. Both land in the same open tables, so one table serves live and historical queries.

## Common questions

### What is change data capture?

Reading a database's change log so inserts, updates and deletes flow to other systems within seconds, without full reloads.

### Does CDC slow down the source database?

Log-based CDC reads the database log rather than querying tables, so the load is small compared with repeated full extracts.

### Can we get exactly-once data?

Writes into tables are transactional and idempotent; duplicates from at-least-once capture are removed by key.

## Implementation details

1. **Pick the hot-path tables.** List the tables and events where seconds matter (transactions, payee changes, network events). Keep everything else on Connect batch loads.
2. **Enable log-based capture** on the source database with a read-only replication account, agreed with the DBA.
3. **Register event contracts** in the StreamHouse schema registry before producers go live, and evolve them compatibly.
4. **Key topics by the decision entity** (card, account, subscriber) so order is preserved per key; size partitions for peak throughput.
5. **Add in-flight processing** for short-window features and enrichment, with event-time windows and an explicit lateness limit.
6. **Land in Bronze Iceberg** with exactly-once, de-duplicated writes; model Silver and Gold with dbt as for batch data.
7. **Write decisions back**: scores, rule hits and actions go to Iceberg next to the event.
8. **Reconcile daily** against the batch path to detect dropped or duplicated events.

Typical hot-path workloads: card and payment fraud, account takeover, wallet transfers, telecom SIM-box detection, network alarms, stock-out alerts and production-line drift. See [StreamHouse: the real-time hot path](/blog/streamhouse-real-time-hot-path-lakehouse).
