Solution blueprint

Real-time CDC lakehouse with StreamHouse

A real-time CDC lakehouse on DataNature captures inserts, updates and deletes from source databases and streams events through managed Kafka into Iceberg tables. StreamHouse provides a schema registry and transactional, idempotent writes, so fraud models, alerts and dashboards work on data that is seconds old.

Updated

How it is built

  1. Capture changes
  2. Stream and validate
  3. Write exactly once
  4. Serve live and history
Connect CDCRow-level change capture from Postgres, Oracle, SQL Server and MySQL logs
StreamHouseManaged Kafka, schema registry, in-flight processing, transactional writes
LakehouseReal-time Bronze tables in Delta or Iceberg
AtlasLineage and access control on streaming data
  1. Capture changesCDC reads database logs and publishes each change as an event.
  2. Stream and validateEvents pass through managed Kafka with schema contracts and in-flight processing.
  3. Write exactly onceKafka transactions and checkpointed, idempotent table commits give exactly-once writes into tables. Source capture is at-least-once; duplicates are removed by key on write.
  4. Serve live and historyThe same tables serve real-time scoring, SQL and model training.
Connect CDC
Row-level change capture from Postgres, Oracle, SQL Server and MySQL logs
StreamHouse
Managed Kafka, schema registry, in-flight processing, transactional writes
Lakehouse
Real-time Bronze tables in Delta or Iceberg
Atlas
Lineage and access control on streaming data
On this page

When should we use real-time instead of batch?

When latency matters: fraud, transaction monitoring, network events, IoT. Batch stays simpler for scheduled loads and reconciliation.

What makes streaming hard, and how is it handled?

Joins, deduplication and late or out-of-order events. StreamHouse preserves ordering per key, removes duplicates on write, and checkpoints track what has been processed.

Can streaming and batch share tables?

Yes. Both land in the same open tables, so one table serves live and historical queries.

Common questions

What is change data capture?

Reading a database's change log so inserts, updates and deletes flow to other systems within seconds, without full reloads.

Does CDC slow down the source database?

Log-based CDC reads the database log rather than querying tables, so the load is small compared with repeated full extracts.

Can we get exactly-once data?

Writes into tables are transactional and idempotent; duplicates from at-least-once capture are removed by key.

Implementation details

  1. Pick the hot-path tables. List the tables and events where seconds matter (transactions, payee changes, network events). Keep everything else on Connect batch loads.
  2. Enable log-based capture on the source database with a read-only replication account, agreed with the DBA.
  3. Register event contracts in the StreamHouse schema registry before producers go live, and evolve them compatibly.
  4. Key topics by the decision entity (card, account, subscriber) so order is preserved per key; size partitions for peak throughput.
  5. Add in-flight processing for short-window features and enrichment, with event-time windows and an explicit lateness limit.
  6. Land in Bronze Iceberg with exactly-once, de-duplicated writes; model Silver and Gold with dbt as for batch data.
  7. Write decisions back: scores, rule hits and actions go to Iceberg next to the event.
  8. Reconcile daily against the batch path to detect dropped or duplicated events.

Typical hot-path workloads: card and payment fraud, account takeover, wallet transfers, telecom SIM-box detection, network alarms, stock-out alerts and production-line drift. See StreamHouse: the real-time hot path.

See it on your own data.

Book a demo

Demos and quotes are handled by Dlytica, the company behind DataNature.