Architecture at scale

Pro lesson Expert

Warehouses, lakes and lakehouses

Compare the architectures companies actually run, see why a folder of files is not a table, and pick the right one for the job.

10 min8 hands-on steps

In this lesson you will

  • Say what a warehouse, a lake and a lakehouse each give you, and what each costs
  • See why a folder of files is not a table, and how a manifest of snapshots fixes it
  • Read a table as it was in the past with time travel
  • Choose an architecture by size and need, not by hype

Words you'll meet

Data warehouse
A managed database for analytics: you load data in, and it gives you fast SQL and safe, all-or-nothing changes.
Data lake
Files in cheap storage, in any format, that many tools can read. Just files: no tables, no all-or-nothing changes.
Lakehouse
A lake plus a table format, so files in cheap storage behave like warehouse tables.
Table format
A log kept next to the files that says which files make up the table, like Apache Iceberg or Delta Lake.
Snapshot
One version of a table: the exact list of files that made it up after one change.
Manifest
The list of files in a snapshot.
Time travel
Reading a table as it was at an earlier snapshot.
Medallion
Naming the layers bronze (raw), silver (clean) and gold (modeled).

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1A warehouse
  2. 2A lake
  3. 3Which files are current?
  4. 4A table format
  5. 5Time travel
  6. 6Commit a snapshot
  7. 7Bronze, silver, gold
  8. 8Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.