Your journey
Curious
100 XP to Trainee
Up next · Foundations
Why data engineering exists
Progress is kept on this device. Create a free account to save it and join the leaderboard.
Foundations
BeginnerWhat data engineering is, how tables work, and your first pipeline. No experience needed.
- 1
Why data engineering exists
No experience needed. Feel the problem first, then meet the tool that solves it.
6 min7 stepsStart - 2
Tables, rows & columns
Meet the shape all data engineering revolves around, then write your first query.
6 min7 steps - 3
Just the rows you need
A manager wants only their shop's orders. Pick rows by a rule, work out totals, and put them in order.
8 min9 steps - 4
From rows to answers
Turn a pile of orders into the numbers the owner asks for, then catch a number that lies.
9 min8 steps - 5
Your first pipeline: raw to clean
Real data arrives dirty. Find the mess, clean it, and load a table people can trust.
10 min7 steps - 6
Loads you can run twice
Jobs crash and get retried. Learn the property that keeps a retry from corrupting your data.
8 min6 steps
The tools nobody teaches
All levelsGit, the command line, Docker, logs and query plans: the everyday tools of data work that courses skip. Real git, plus practice versions of the terminal and Docker, all running in your browser.
- 7
What git actually is
Not a backup button. Open it up and see the snapshots, fingerprints and objects inside, in a real terminal.
12 min7 steps Commits, history and time travel
ProRead what changed, commit only what is ready, visit old versions, and undo mistakes without panic.
14 min8 stepsBranches are just pointers
ProA branch is one tiny file holding a hash. See it, make branches, work in parallel and merge it all back.
13 min7 stepsMerge conflicts without panic
ProWhen two people change the same lines, git stops and asks you. Read a conflict, settle it, and finish the merge.
12 min8 steps- 11
The command line for data
Look inside huge files without opening them, count, filter and group from the terminal, and pull errors out of a log.
12 min7 steps When a pipeline fails at 2am
ProAn incident drill: read the logs, find the first error and the row behind it, set the bad rows aside, and plan the backfill.
15 min9 stepsSomeone else's SQL
ProFind your way around a model you did not write: read it in the right order, test each step on its own, find the join that duplicates rows, and prove your fix.
15 min7 stepsDocker for data engineers
ProPackage a pipeline so it runs the same on your laptop, a server and the cloud. Build images, run containers, and make rebuilds fast.
15 min8 stepsWhy is this query slow?
ProWhat a query plan is, how to read one from the bottom up, and how to find the step that is costing you.
12 min9 steps
Combining data
BeginnerReal data is spread across many tables. Join it without losing or doubling anything.
- 16
Joining tables
Real data lives in many tables. Learn to stitch them together, and to notice when rows go missing.
10 min7 steps When joins multiply money
ProJoin two tables at different grains and totals quietly inflate. Learn to spot it and fix it.
10 min7 stepsStacking batches
ProCombine files that arrive day by day with UNION, and keep their columns lined up.
10 min9 steps
SQL you'll use every day
BeginnerThe SQL that trips people up at work: missing values, buckets, dates and time zones, readable queries, and the days with no data.
NULL, the missing value
Coming soonWhy NOT IN can return nothing, why COUNT(*) and COUNT(column) disagree, and how to fill the gaps.
10 minSorting rows into buckets
Coming soonLabel, group and count rows by rules of your own with CASE WHEN.
10 minDates, times and time zones
Coming soonTruncate to the day, week or month, and stop midnight in one time zone landing on the wrong day in another.
10 minReadable SQL with WITH
Coming soonBreak a long query into named steps that you, and the next person, can follow.
10 minRows into columns
ProComing soonTurn one row per month into one column per month, for the reports people ask for.
10 minThe days with no sales
ProComing soonA day with no rows vanishes from a chart. Build a calendar and keep every day in.
10 min
Reliable pipelines
IntermediatePipelines that run every night without waking anyone up: tests, incremental loads and scheduling.
Taming messy text
ProMatch values that differ in case, spacing and spelling: LOWER, TRIM, LIKE and lookup tables.
10 min9 stepsData quality tests
ProWrite the checks that catch bad data before your users do.
10 min9 stepsLatest wins: window functions
ProKeep the newest version of each row when updates arrive more than once.
10 min8 stepsIncremental loads
ProLoad only what changed since last time, without losing the rows that arrive late.
10 min8 stepsOrchestration
ProChain jobs into a DAG, handle retries, and backfill history.
10 min8 steps
Modeling for analytics
IntermediateShape data so analysts get the right answer by default: facts, dimensions and history.
Files, formats & storage
AdvancedWhere data actually lives: CSV, JSON, Parquet, and how to lay files out so queries fly.
Architecture at scale
ExpertThe decisions senior engineers make: batch or streaming, warehouse or lake, speed against cost.
Batch and streaming
ProProcess data every night or every second: windows, late events, duplicates, and when streaming is worth what it costs.
10 min8 stepsWarehouses, lakes and lakehouses
ProCompare the architectures companies actually run, see why a folder of files is not a table, and pick the right one for the job.
10 min8 stepsPerformance and cost
ProFind the query that is costing you money, make it read less, and prove the cheaper answer is still right.
10 min9 steps
Capstone: your own pipeline
ExpertBuild a working pipeline front to back, from raw files to a trusted dashboard, for the coffee chain you have followed all along.
Land the raw data
ProCopy the shops' messy till files into raw tables, untouched, with a load you can safely run twice.
20 min11 stepsClean and model it
ProTurn raw orders into tested, deduplicated fact and dimension tables.
25 min11 stepsMake it run itself
ProWire the steps into a DAG with incremental loads, tests and retries.
25 min11 stepsShip the dashboard
ProServe the numbers the owner asked for on day one, and prove they are right.
20 min10 steps
Interview drills
IntermediateThe SQL questions data engineering interviews keep asking, as short timed drills with the traps interviewers look for.
The second-highest value
ProComing soonThe classic opener, with the ties and empty results that catch most answers out.
8 minTop N per group
ProComing soonThe best three of everything: per store, per month, per customer.
8 minRunning totals and moving averages
ProComing soonTotals that build up row by row, and averages over the last seven days.
8 minSplitting clicks into sessions
ProComing soonGroup a stream of clicks into visits, ending one after 30 quiet minutes.
8 minStreaks and gaps
ProComing soonFind the longest run of consecutive days, the gaps-and-islands question.
8 minCohort retention
ProComing soonOf the customers who first ordered in March, how many came back each month after?
8 minFind the bug
ProComing soonEach query runs and returns a number. The number is wrong. Say why, then fix it.
8 min