Modern Data Lakehouse
Module 20 · Core
Medallion architecture
Bronze, silver and gold: what each layer promises, and how to build pipelines you can rerun safely.
In plain words
Most pipelines move data through three layers: bronze keeps it raw, silver cleans and standardises it, gold shapes it for the business. This module follows one messy record through all three, and shows how to build pipelines you can safely rerun.
Helps to take first
Words you'll meet
This module is in production
Here's what it will cover.
Centrepiece
Follow one messy record from bronze to gold, then wire the layers into a pipeline and replay a bad day of data
You will understand
- Layer contracts and data quality expectations
- Idempotency, backfills and reprocessing
- Transformation tools: dbt, Spark, Lakeflow/DLT
- Orchestration: Airflow, Dagster
Formats
Scroll storyBuild & connectFix the problemCheckpoints