Glossary

Every Modern Data Lakehouse term in plain words, with the module that teaches it. Inside modules, any word with a dotted underline explains itself.

Modern Data Lakehouse · 95 terms

Apache Arrow: A columnar format for data in memory, so engines and languages can share data without converting it. It complements Parquet, which is for storage.; Learn it in “Rows vs columns: file formats” →
Apache Hudi: An open table format built for fast upserts and incremental processing, organised around a timeline of actions.; Learn it in “Apache Hudi: the timeline” →
Apache Iceberg: An open table format that tracks files through a tree of metadata, with the current state held by a catalog pointer.; Learn it in “Apache Iceberg: the metadata tree” →
Apache XTable: An incubating Apache project (formerly OneTable) that translates table metadata between Delta, Iceberg and Hudi, without copying data files.; Learn it in “Delta vs Iceberg vs Hudi” →
Avro: A compact, row-oriented binary format with an embedded schema. Popular for streaming and data exchange.; Learn it in “Rows vs columns: file formats” →
Blind append: A write that adds new rows without reading the table first, such as a plain INSERT. It can't be invalidated by other writers' data changes.; Learn it in “ACID on object storage” →
Bloom filter: A compact structure that answers “is this value definitely not here?”. Useful for skipping data on equality filters like id = 42 when min/max can't help.; Learn it in “Inside a Parquet file” →
Branch / tag (Iceberg): Named pointers to snapshots. A branch can take new commits (e.g. for write-audit-publish); a tag is fixed and can keep a snapshot from being expired.; Learn it in “Apache Iceberg: the metadata tree” →
Bucket: A top-level container in object storage. Every object lives in exactly one bucket.; Learn it in “Object storage: the ground floor” →
Catalog: The directory of all tables: their names, where their metadata lives and who may access them. Engines ask the catalog first.; Think of it as: A library catalogue: it doesn't hold the books, but tells you exactly where each one is.; Learn it in “Catalogs: the source of truth” →
CDC: Change data capture: turning every insert, update and delete in a database into a stream of change events.; Learn it in “CDC, MERGE & slowly changing dimensions” →
Checkpoint (Delta): A Parquet file that summarises the whole table state at one version, so readers don't have to replay every log entry from the start.; Learn it in “Delta Lake: the transaction log” →
Column chunk: All the values of one column within one row group, stored together in one contiguous stretch of a Parquet file.; Learn it in “Inside a Parquet file” →
Column mapping (Delta): A Delta table feature that gives columns stable IDs/physical names, so renames and drops become metadata-only changes. Set with delta.columnMapping.mode = 'name'.; Learn it in “Schema evolution & enforcement” →
Columnar storage: Storing all values of one column together, instead of one row at a time. Analytics queries then read only the columns they need.; Think of it as: A cupboard sorted by item type (all cups together) instead of by meal.; Learn it in “Rows vs columns: file formats” →
Commit: The moment a change becomes official and visible. In a table format, writing the next log or metadata entry.; Learn it in “Delta Lake: the transaction log” →
Compaction: Rewriting many small files into fewer large ones (OPTIMIZE in Delta, rewrite_data_files in Iceberg).; Learn it in “Keeping tables healthy” →
Compression: Encoding data in fewer bytes. Columnar data compresses very well because similar values sit together.; Learn it in “Inside a Parquet file” →
Copy-on-write: Changing rows by rewriting the whole files that contain them. Reads stay simple; writes are expensive.; Learn it in “Updates & deletes: Copy-on-Write vs Merge-on-Read” →
Data lake: A large, cheap store that keeps data of every kind in its raw form, usually as files in object storage.; Learn it in “From swamp to lakehouse” →
Data skipping: Using metadata and statistics to avoid reading files or chunks that can't contain matching rows.; Learn it in “File layout, clustering & data skipping” →
Data swamp: A data lake nobody trusts any more: no transactions, no enforced schema and no catalog, so nobody knows which data is right.; Learn it in “From swamp to lakehouse” →
Data warehouse: A database built for analytics: clean tables and fast SQL, traditionally with storage and compute bundled in one system.; Learn it in “From swamp to lakehouse” →
Deletion vector: A small bitmap marking which rows of a file are deleted, so the file doesn't have to be rewritten.; Learn it in “Updates & deletes: Copy-on-Write vs Merge-on-Read” →
Delta Lake: An open table format that records every change in an ordered transaction log (_delta_log/).; Learn it in “Delta Lake: the transaction log” →
Encoding: Clever ways of writing values compactly before compression, such as a dictionary (store 'paid' once, then refer to it) or run-length (write '5 × paid').; Learn it in “Inside a Parquet file” →
Equality delete: An Iceberg delete file that removes rows by column values (e.g. order_id = 1042). Cheap to write; readers must check older rows against it.; Learn it in “Apache Iceberg: the metadata tree” →
ETL: Extract, transform, load: a job that copies data out of one system, reshapes it, and loads it into another.; Learn it in “From swamp to lakehouse” →
Field ID: A permanent number Iceberg gives every column. Data files are read by ID, not name, so renames, drops and re-adds can never mix up columns.; Learn it in “Apache Iceberg: the metadata tree” →
File group: Hudi's unit of storage within a partition: a set of records identified by a file ID, stored as a series of file slices (a base file plus log files).; Learn it in “Apache Hudi: the timeline” →
Hidden partitioning: Iceberg derives partition values from ordinary columns with a transform (e.g. days(order_ts)), so users never fill in or filter on a separate partition column.; Learn it in “Apache Iceberg: the metadata tree” →
Hilbert curve: A space-filling curve that visits every cell of a grid while only ever stepping to a neighbouring cell. Ordering rows along it keeps rows with similar values in several columns in the same files.; Learn it in “File layout, clustering & data skipping” →
Hive Metastore: A long-standing catalog service that stores each table's name, schema, partitions and folder location. It stores metadata only; the data files live in storage.; Learn it in “What makes a table a table” →
Hive-style table: A table defined as “all the files in this folder”, usually with one sub-folder per partition value (date=2026-09-24/). Simple, but with no record of which files belong at any moment.; Learn it in “What makes a table a table” →
Index (Hudi): The mechanism Hudi uses to find which file group holds a record key, such as Simple, Bloom, Bucket or a record-level index in the metadata table.; Learn it in “Apache Hudi: the timeline” →
Instant (Hudi): One action on the Hudi timeline: an action type, a requested time, a state (requested, inflight or completed) and, once done, a completion time.; Learn it in “Apache Hudi: the timeline” →
Isolation: How far concurrent transactions are shielded from each other's half-finished work.; Learn it in “ACID on object storage” →
Key and prefix: An object's key is its full name, like sales/2026/09/part-0.parquet. A prefix is the start of a key; the “folders” you see are just prefixes.; Learn it in “Object storage: the ground floor” →
Lakehouse: An architecture that gives warehouse-style guarantees (transactions, schema, governance) on one copy of open files in object storage, readable by many engines.; Learn it in “From swamp to lakehouse” →
Lifecycle rule: A rule that moves objects to cheaper storage classes, or deletes them, once they reach a certain age.; Learn it in “Object storage: the ground floor” →
Liquid clustering: A Delta Lake layout that groups rows into files by up to four clustering keys instead of partition folders. Keys can change without rewriting existing data.; Learn it in “Partitioning done right” →
Manifest: An Avro file listing a set of Iceberg data (or delete) files, with each file's partition, row count, size and per-column statistics such as min and max.; Learn it in “Apache Iceberg: the metadata tree” →
Manifest list: An Avro file, one per Iceberg snapshot, naming the manifests in that snapshot with a summary of each manifest's partition values, so whole manifests can be skipped.; Learn it in “Apache Iceberg: the metadata tree” →
Medallion architecture: Organising pipelines in layers: bronze (raw), silver (cleaned and conformed) and gold (business-ready).; Learn it in “Medallion architecture” →
MERGE: A SQL statement that upserts and deletes rows in one go by matching a source against a target table.; Learn it in “CDC, MERGE & slowly changing dimensions” →
Merge-on-read: Recording changes separately (delete files, logs) and merging them in when reading. Writes are cheap; reads do extra work.; Learn it in “Updates & deletes: Copy-on-Write vs Merge-on-Read” →
Metadata file (Iceberg): A JSON file describing an Iceberg table: its schemas, partition specs, snapshots and which snapshot is current. Each commit writes a new one; the catalog points to the latest.; Think of it as: The library's guide, reprinted as a new edition for every change.; Learn it in “Apache Iceberg: the metadata tree” →
Metadata table (Hudi): An internal table under .hoodie/metadata that stores file listings, column statistics and indexes, so engines don't have to list storage or open every file.; Learn it in “Apache Hudi: the timeline” →
Min/max statistics: The smallest and largest value of a column in a file or row group. If you look for amount > 500 and the max is 300, the whole chunk can be skipped.; Learn it in “Inside a Parquet file” →
Object storage: Cloud storage (Amazon S3, Google Cloud Storage, Azure) that stores each blob of bytes under a key. Cheap, durable and huge, but not a real file system.; Think of it as: A giant cloakroom: hand over a coat, get a ticket (the key), and get the coat back with the ticket.; Learn it in “Object storage: the ground floor” →
OLAP: Online analytical processing: big scans and summaries over lots of historical data, like revenue by city for three years.; Think of it as: The accountant at month end, going through every bill in the shop.; Learn it in “From swamp to lakehouse” →
OLTP: Online transaction processing: many small, fast reads and writes of individual records, like placing an order or updating stock.; Think of it as: The billing counter: one customer at a time, and it must be quick.; Learn it in “From swamp to lakehouse” →
Open table format: A layer of metadata over data files (Delta Lake, Apache Iceberg, Apache Hudi) that turns a folder of files into a real table with transactions.; Learn it in “What makes a table a table” →
Optimistic concurrency: Letting writers work in parallel without locks, and checking for conflicts only at commit time. On a clash, one retries or fails.; Learn it in “ACID on object storage” →
ORC: Optimized Row Columnar: an open columnar file format from the Hive world, similar in spirit to Parquet.; Learn it in “Rows vs columns: file formats” →
Orphan files: Files in a table's storage that no version of the table references, usually left by failed or aborted writes. Invisible to queries, but still stored and billed.; Learn it in “Keeping tables healthy” →
Page index: Optional per-page min/max values and locations, stored near the footer, so a reader can skip individual pages inside a column chunk.; Learn it in “Inside a Parquet file” →
Parquet: The most widely used open columnar file format for analytics. It stores data column by column, compressed, with statistics that let engines skip data.; Learn it in “Inside a Parquet file” →
Parquet footer: The metadata at the end of a Parquet file: the schema, where every row group and column chunk sits, and statistics such as min and max. Readers read it first.; Think of it as: The index at the back of a book.; Learn it in “Inside a Parquet file” →
Partition: Splitting a table into groups by a column (such as date) so queries that filter on it only read the matching groups.; Think of it as: A file cabinet with one drawer per month.; Learn it in “Partitioning done right” →
Partition evolution: Changing how a table is partitioned as a metadata-only change. New data uses the new layout; old files keep theirs and are never rewritten.; Learn it in “Apache Iceberg: the metadata tree” →
Partition pruning: Skipping whole partitions (folders) whose values can't match a query's filter on the partition column.; Think of it as: Opening only the drawer labelled with the date you need.; Learn it in “Partitioning done right” →
Partition transform: A function that turns a column value into a partition value: identity, years, months, days, hours, bucket(N) or truncate(W).; Learn it in “Apache Iceberg: the metadata tree” →
Position delete: An Iceberg delete file that lists a data file's path and the row positions to skip. Replaced by deletion vectors in format v3.; Learn it in “Apache Iceberg: the metadata tree” →
Predicate pushdown: Applying a query's filter (the WHERE clause) as early as possible, at the file or storage level, so less data is read.; Learn it in “How engines read a lakehouse” →
Projection pruning: Reading only the columns a query actually uses and skipping the rest.; Learn it in “Inside a Parquet file” →
Put-if-absent: A write that only succeeds if the key doesn't exist yet. When two writers race, exactly one wins. Also called a conditional write.; Learn it in “Object storage: the ground floor” →
Query engine: The software that actually runs your SQL or code over the data, such as Spark, Trino, DuckDB or a cloud warehouse.; Think of it as: The chef. The data is the pantry.; Learn it in “How engines read a lakehouse” →
Query plan: The step-by-step recipe an engine builds to run a query: which files to read, how to filter, join and aggregate.; Learn it in “How engines read a lakehouse” →
Read amplification: Extra work every read must do beyond reading the data itself, e.g. applying pending delete files, deletion vectors or log files under Merge-on-Read.; Learn it in “Updates & deletes: Copy-on-Write vs Merge-on-Read” →
Record key: The field (or fields) that uniquely identify a record, such as trip_id. Hudi uses it to route updates and deletes to the right file group.; Learn it in “Apache Hudi: the timeline” →
Repetition & definition levels: Two small numbers stored with each value of a nested column, recording where lists start and where values are missing, so nested data can be stored column by column.; Learn it in “Inside a Parquet file” →
Retention period: How long old files or history are kept before clean-up may delete them. It limits how far back time travel reaches.; Learn it in “Keeping tables healthy” →
Row group: A horizontal slice of a Parquet file, typically many thousands of rows, stored column by column.; Learn it in “Inside a Parquet file” →
Schema enforcement: Rejecting writes that don't match the table's schema (wrong types, unknown columns, missing required values). The whole write fails; nothing is committed.; Think of it as: A receptionist who won't accept a form with letters in the phone-number box.; Learn it in “Schema evolution & enforcement” →
Schema evolution: Changing a table's schema on purpose (adding, renaming, dropping or widening columns) without breaking old data or rewriting it.; Learn it in “Schema evolution & enforcement” →
Schema-on-read: Storing data as-is and deciding its structure only when reading it. Flexible, but nothing stops bad data from coming in.; Learn it in “From swamp to lakehouse” →
Self-describing file: A file that carries its own schema (column names and types) inside it, so a reader needs nothing else to understand it. Avro, Parquet and ORC are; CSV is not.; Learn it in “Rows vs columns: file formats” →
Serializable isolation: The strictest common isolation level: concurrent transactions must produce a result that matches running them one at a time, in the order the history shows.; Learn it in “ACID on object storage” →
Serialization: Turning data in memory (like a table) into a line of bytes that can be saved to a file or sent over a network, and back again.; Think of it as: Packing a room into boxes for a move, in an order you can unpack from.; Learn it in “Rows vs columns: file formats” →
Slowly changing dimension: Ways to handle attributes that change over time, like a customer's city: overwrite it (Type 1) or keep history rows (Type 2).; Learn it in “CDC, MERGE & slowly changing dimensions” →
Small-files problem: When a table is split into too many tiny files, engines spend more time opening files and making requests than reading data.; Learn it in “File layout, clustering & data skipping” →
Snapshot / version: The complete state of a table at one point in its history: exactly which files made it up.; Learn it in “Delta Lake: the transaction log” →
Splittable: A file is splittable if many workers can each start reading from the middle of it, so one big file can be processed in parallel.; Learn it in “Rows vs columns: file formats” →
Storage class: A price and speed tier in object storage. Colder classes cost less to keep but more (or longer) to read.; Learn it in “Object storage: the ground floor” →
Time travel: Querying a table as it was at an earlier version or time, e.g. VERSION AS OF 3.; Learn it in “Delta Lake: the transaction log” →
Timeline (Hudi): Hudi's record of every action on a table (commits, cleans, compactions…), stored as small files under .hoodie/timeline/. Readers only trust actions marked completed.; Think of it as: The order rail in a restaurant kitchen: tickets requested, in progress, served.; Learn it in “Apache Hudi: the timeline” →
Transaction log: An append-only list of every change made to a table, in order. Replaying it gives the table at any point.; Think of it as: A bank passbook: every deposit and withdrawal is a line; the balance is what the lines add up to.; Learn it in “Delta Lake: the transaction log” →
UniForm (Delta): A Delta Lake feature that also writes Iceberg (and, in preview, Hudi) metadata for the same Parquet files, so Iceberg readers can query a Delta table without copying it.; Learn it in “Delta vs Iceberg vs Hudi” →
Upsert: Update the row if it exists, insert it if it doesn't.; Learn it in “CDC, MERGE & slowly changing dimensions” →
VACUUM: A maintenance command that permanently deletes data files no longer used by the table and older than the retention period.; Learn it in “Keeping tables healthy” →
VARIANT type: A column type for semi-structured, JSON-like data stored in an efficient binary encoding, so fields can vary from row to row without schema changes.; Learn it in “Schema evolution & enforcement” →
Write amplification: Writing far more data than actually changed, e.g. rewriting a whole 128 MB file to change one row under Copy-on-Write.; Learn it in “Updates & deletes: Copy-on-Write vs Merge-on-Read” →
WriteSerializable: Databricks' default Delta isolation level. Writes are serializable, but a blind append doesn't conflict with a concurrent reader, so more jobs succeed while history can describe an order that didn't really happen.; Learn it in “ACID on object storage” →
Z-order / clustering: Arranging rows so that values of several columns sit close together in the same files, so more files can be skipped.; Learn it in “File layout, clustering & data skipping” →

General terms · shared by every track

ACID: Four guarantees for transactions: Atomic (all or nothing), Consistent (rules hold), Isolated (concurrent work doesn't interfere), Durable (once done, it stays done).; Learn it in “ACID on object storage” →
Atomic: All or nothing. An atomic change either happens completely or not at all, and nobody ever sees it half done.; Think of it as: Handing over a sealed envelope: the other person has it or doesn't. There's no half-handed envelope.; Learn it in “Object storage: the ground floor” →
Batch processing: Processing data in scheduled chunks, such as every night or every hour.; Learn it in “Getting data in” →
CSV: Comma-separated values: plain text, one row per line. Easy to read, slow and large for analytics, and no schema.; Learn it in “Rows vs columns: file formats” →
Idempotent: Safe to run twice: running the job again gives the same result instead of duplicating data.; Learn it in “Medallion architecture” →
JSON: A text format of nested key-value objects, like {"id": 7}. Common for app events and APIs, and used by Delta's transaction log.; Learn it in “Rows vs columns: file formats” →
Metadata: Data about data: which files exist, their schema, sizes and statistics. Engines read metadata first to plan work.; Learn it in “What makes a table a table” →
Schema: The structure of a table: its column names and types, such as order_id: long, amount: decimal.; Learn it in “Schema evolution & enforcement” →
SQL: The standard language for querying tables: SELECT … FROM … WHERE …; Learn it in “Hands-on: query in your browser” →
Streaming: Processing data continuously as it arrives, within seconds or minutes.; Learn it in “Getting data in” →
Transaction: A group of changes that succeeds or fails as one unit.; Think of it as: A money transfer: debit one account and credit the other, or do neither.; Learn it in “ACID on object storage” →