Modern Data Lakehouse

Why warehouses and data lakes each fell short, and what the lakehouse architecture actually changes.

S3, GCS and ADLS are not file systems. Their guarantees shape everything built on top.

CSV, JSON, Avro, ORC and Parquet, and why analytics wants columns.

Open up a Parquet file layer by layer: row groups, column chunks, pages, encodings and the footer.

A folder of Parquet files is not a table. See how Hive-style tables broke, and what a table format must provide.

Every Delta table is a log of commits. Replay it to get any version of the table.

Catalog pointer, metadata file, manifest list, manifests, data files: follow a query down the tree.

Hudi was built for upserts and incremental processing. See its timeline, table types and indexes.

The same operations in all three formats side by side, where they are converging, and how to interoperate.

How commits stay atomic without a database server, and what happens when two writers race.

Files are immutable, so how do you change a row? Compare the strategies and their costs.

Add, rename, drop and widen columns safely, and stop bad data at the door.

Partitioning can make queries fly or drown the table in tiny files.

Small files, sort orders, Z-order and statistics: how engines avoid reading data.

Compaction, vacuum, snapshot expiry and orphan files: the routine work that keeps cost and latency down.