Your journey

Curious

0 day streak0 XP

100 XP to Trainee

Up next · Foundations

Why data engineering exists

Start

Progress is kept on this device. Create a free account to save it and join the leaderboard.

Track 1

Foundations

Beginner

What data engineering is, how tables work, and your first pipeline. No experience needed.

  1. 1

    Why data engineering exists

    No experience needed. Feel the problem first, then meet the tool that solves it.

    6 min7 stepsStart
  2. 2

    Tables, rows & columns

    Meet the shape all data engineering revolves around, then write your first query.

    6 min7 steps
  3. 3

    Just the rows you need

    A manager wants only their shop's orders. Pick rows by a rule, work out totals, and put them in order.

    8 min9 steps
  4. 4

    From rows to answers

    Turn a pile of orders into the numbers the owner asks for, then catch a number that lies.

    9 min8 steps
  5. 5

    Your first pipeline: raw to clean

    Real data arrives dirty. Find the mess, clean it, and load a table people can trust.

    10 min7 steps
  6. 6

    Loads you can run twice

    Jobs crash and get retried. Learn the property that keeps a retry from corrupting your data.

    8 min6 steps
Track 2

The tools nobody teaches

All levels

Git, the command line, Docker, logs and query plans: the everyday tools of data work that courses skip. Real git, plus practice versions of the terminal and Docker, all running in your browser.

  1. 7

    What git actually is

    Not a backup button. Open it up and see the snapshots, fingerprints and objects inside, in a real terminal.

    12 min7 steps
  2. Commits, history and time travel

    Pro

    Read what changed, commit only what is ready, visit old versions, and undo mistakes without panic.

    14 min8 steps
  3. Branches are just pointers

    Pro

    A branch is one tiny file holding a hash. See it, make branches, work in parallel and merge it all back.

    13 min7 steps
  4. Merge conflicts without panic

    Pro

    When two people change the same lines, git stops and asks you. Read a conflict, settle it, and finish the merge.

    12 min8 steps
  5. 11

    The command line for data

    Look inside huge files without opening them, count, filter and group from the terminal, and pull errors out of a log.

    12 min7 steps
  6. When a pipeline fails at 2am

    Pro

    An incident drill: read the logs, find the first error and the row behind it, set the bad rows aside, and plan the backfill.

    15 min9 steps
  7. Someone else's SQL

    Pro

    Find your way around a model you did not write: read it in the right order, test each step on its own, find the join that duplicates rows, and prove your fix.

    15 min7 steps
  8. Docker for data engineers

    Pro

    Package a pipeline so it runs the same on your laptop, a server and the cloud. Build images, run containers, and make rebuilds fast.

    15 min8 steps
  9. Why is this query slow?

    Pro

    What a query plan is, how to read one from the bottom up, and how to find the step that is costing you.

    12 min9 steps
Track 3

Combining data

Beginner

Real data is spread across many tables. Join it without losing or doubling anything.

  1. 16

    Joining tables

    Real data lives in many tables. Learn to stitch them together, and to notice when rows go missing.

    10 min7 steps
  2. When joins multiply money

    Pro

    Join two tables at different grains and totals quietly inflate. Learn to spot it and fix it.

    10 min7 steps
  3. Stacking batches

    Pro

    Combine files that arrive day by day with UNION, and keep their columns lined up.

    10 min9 steps
Track 4

SQL you'll use every day

Beginner

The SQL that trips people up at work: missing values, buckets, dates and time zones, readable queries, and the days with no data.

  1. NULL, the missing value

    Coming soon

    Why NOT IN can return nothing, why COUNT(*) and COUNT(column) disagree, and how to fill the gaps.

    10 min
  2. Sorting rows into buckets

    Coming soon

    Label, group and count rows by rules of your own with CASE WHEN.

    10 min
  3. Dates, times and time zones

    Coming soon

    Truncate to the day, week or month, and stop midnight in one time zone landing on the wrong day in another.

    10 min
  4. Readable SQL with WITH

    Coming soon

    Break a long query into named steps that you, and the next person, can follow.

    10 min
  5. Rows into columns

    ProComing soon

    Turn one row per month into one column per month, for the reports people ask for.

    10 min
  6. The days with no sales

    ProComing soon

    A day with no rows vanishes from a chart. Build a calendar and keep every day in.

    10 min
Track 5

Reliable pipelines

Intermediate

Pipelines that run every night without waking anyone up: tests, incremental loads and scheduling.

  1. Taming messy text

    Pro

    Match values that differ in case, spacing and spelling: LOWER, TRIM, LIKE and lookup tables.

    10 min9 steps
  2. Data quality tests

    Pro

    Write the checks that catch bad data before your users do.

    10 min9 steps
  3. Latest wins: window functions

    Pro

    Keep the newest version of each row when updates arrive more than once.

    10 min8 steps
  4. Incremental loads

    Pro

    Load only what changed since last time, without losing the rows that arrive late.

    10 min8 steps
  5. Orchestration

    Pro

    Chain jobs into a DAG, handle retries, and backfill history.

    10 min8 steps
Track 6

Modeling for analytics

Intermediate

Shape data so analysts get the right answer by default: facts, dimensions and history.

  1. Facts and dimensions

    Pro

    Separate what happened from what it happened to.

    10 min9 steps
  2. Star schemas

    Pro

    Build the model most data warehouses are organized around.

    10 min9 steps
  3. Tracking history

    Pro

    Slowly changing dimensions: remember what was true at the time.

    10 min9 steps
Track 7

Files, formats & storage

Advanced

Where data actually lives: CSV, JSON, Parquet, and how to lay files out so queries fly.

  1. Reading CSV and JSON

    Pro

    Load messy real-world files and handle what goes wrong.

    10 min8 steps
  2. Parquet and columnar storage

    Pro

    Why analytics files are stored by column, and what it buys you.

    10 min7 steps
  3. Partitioning

    Pro

    Split big data so queries only read what they need.

    10 min8 steps
Track 8

Architecture at scale

Expert

The decisions senior engineers make: batch or streaming, warehouse or lake, speed against cost.

  1. Batch and streaming

    Pro

    Process data every night or every second: windows, late events, duplicates, and when streaming is worth what it costs.

    10 min8 steps
  2. Warehouses, lakes and lakehouses

    Pro

    Compare the architectures companies actually run, see why a folder of files is not a table, and pick the right one for the job.

    10 min8 steps
  3. Performance and cost

    Pro

    Find the query that is costing you money, make it read less, and prove the cheaper answer is still right.

    10 min9 steps
Track 9

Capstone: your own pipeline

Expert

Build a working pipeline front to back, from raw files to a trusted dashboard, for the coffee chain you have followed all along.

  1. Land the raw data

    Pro

    Copy the shops' messy till files into raw tables, untouched, with a load you can safely run twice.

    20 min11 steps
  2. Clean and model it

    Pro

    Turn raw orders into tested, deduplicated fact and dimension tables.

    25 min11 steps
  3. Make it run itself

    Pro

    Wire the steps into a DAG with incremental loads, tests and retries.

    25 min11 steps
  4. Ship the dashboard

    Pro

    Serve the numbers the owner asked for on day one, and prove they are right.

    20 min10 steps
Track 10

Interview drills

Intermediate

The SQL questions data engineering interviews keep asking, as short timed drills with the traps interviewers look for.

  1. The second-highest value

    ProComing soon

    The classic opener, with the ties and empty results that catch most answers out.

    8 min
  2. Top N per group

    ProComing soon

    The best three of everything: per store, per month, per customer.

    8 min
  3. Running totals and moving averages

    ProComing soon

    Totals that build up row by row, and averages over the last seven days.

    8 min
  4. Splitting clicks into sessions

    ProComing soon

    Group a stream of clicks into visits, ending one after 30 quiet minutes.

    8 min
  5. Streaks and gaps

    ProComing soon

    Find the longest run of consecutive days, the gaps-and-islands question.

    8 min
  6. Cohort retention

    ProComing soon

    Of the customers who first ordered in March, how many came back each month after?

    8 min
  7. Find the bug

    ProComing soon

    Each query runs and returns a number. The number is wrong. Say why, then fix it.

    8 min