Architecture at scale

Pro lesson Expert

Performance and cost

Find the query that is costing you money, make it read less, and prove the cheaper answer is still right.

10 min9 hands-on steps

In this lesson you will

  • Work out what each query costs, and which one to fix first
  • Cut the bytes a query scans: fewer columns, partition filters, summary tables
  • Keep a summary table fresh incrementally, and prove it matches the source
  • Know when an approximate count is good enough

Words you'll meet

Bytes scanned
How much data a query has to read. Many cloud warehouses bill by it.
Pre-aggregation
Adding up the data ahead of time, so queries read the totals instead of every row.
Summary table
A table of pre-aggregated totals, like one row per day per city.
Materialize
Save a result as a real table so it is computed once and read many times.
Approximate count
A count that is usually close, in return for far less memory and time.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1Where the money goes
  2. 2Pay for the columns you read
  3. 3Measure first
  4. 4Pre-aggregate
  5. 5Measure again
  6. 6Keep it fresh
  7. 7Close enough
  8. 8Filter on the partition
  9. 9Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.