Modern Data Lakehouse
Module 19 · Deep dive
CDC, MERGE & slowly changing dimensions
Turn a stream of database changes into a correct, up-to-date lakehouse table.
In plain words
When an app database changes, those changes can be streamed as events: insert this, update that, delete this. Applying them to a lakehouse table correctly, even when they arrive late or twice, is the job of MERGE. You'll also learn how to keep history of changes.
Helps to take first
Words you'll meet
This module is in production
Here's what it will cover.
Centrepiece
Replay Debezium change events into a table with MERGE; inject out-of-order and duplicate events and fix the result
You will understand
- Change data capture with Debezium, DMS and Datastream
- MERGE INTO semantics, ordering and deduplication
- SCD Type 1 and Type 2
- Change feeds: Delta CDF, Iceberg changelog, Hudi incremental queries
Formats
Step-throughFix the problemCheckpoints