Free lesson

Why data engineering exists

No experience needed. Feel the problem first, then meet the tool that solves it.

6 min7 steps

In this lesson you will

  • Why companies need data engineers
  • What a table and a pipeline are
  • Write your first line of SQL

Words you'll meet

Table
Data arranged in rows and columns, one fact per box.
Pipeline
Software that moves data from where it is made to where it is useful, with nobody touching it.
SQL
The language used to ask questions of tables.
Start the lesson

The lesson, step by step

Each step teaches one idea, then hands you a real database to try it on. The exercises run in your browser.

  1. Step 1 of 7

    A simple question

    Every business runs on records of what happened: sales, clicks, deliveries, payments. That collection of records is its data.

    A small coffee chain writes down every order. Here are this week's. Each line is one order: where it was sold and how many drinks.

    Your turn: Which city sold the most drinks? Click any order from that city.

  2. Step 2 of 7

    The same question, bigger

    Data grows much faster than anyone can read it. A busy shop makes thousands of records a week; a website makes millions a day.

    Business is good. The list now has 8,000 orders, and the owner asks you the same thing.

    Your turn: Which city sold the most drinks?

  3. Step 3 of 7

    Ask the computer

    A computer can do it in a blink, if you ask in a language it understands. The most common one for questions about data is called SQL.

    The same question is written in SQL below. You do not need to understand it yet. Soon you will write ones like it yourself.

    SQL has been around since the 1970s and almost every database speaks it. People say it as "sequel" or spell it out; both are fine.

    The query here reads nearly like English: take the city and the total drinks, from the big table, one line per city, biggest first. It is the one skill every data engineering job asks for.

    Your turn: Press Run.

  4. Step 4 of 7

    Do it by hand

    That only worked because the orders were already in a neat grid, one fact per box. A grid like that is called a table.

    Orders do not start out that way. They start as scribbles. Somebody has to file each one into the table. For the next minute, that somebody is you.

    Each column holds one kind of fact and each row holds one order. Once data is in that shape, a computer can count it, add it up and sort it.

    Turning scribbles into tables is called ingestion, and it is the first stage of nearly every pipeline. The scribbles might be text, web forms, clicks in an app or readings from a sensor.

    Your turn: File each order slip into the table, one part at a time.

  5. Step 5 of 7

    Connect the steps

    A pipeline is a row of steps. Data goes in one end and something useful comes out the other, with nobody touching it in between.

    Real pipelines have the same stages under grander names: sources, transformation, storage (often called a data warehouse) and reporting. You will also hear ETL, short for extract, transform, load.

    Most pipelines run on a schedule, such as every night. Some run all the time, handling each order the moment it happens.

    Your turn: Click the four steps in the order the data should flow.

  6. Step 6 of 7

    When a step fails

    The next morning the owner looks at the chart and frowns. Austin has two bars.

    Pipelines rarely fail loudly. More often they keep running and produce a wrong number. That is worse than a crash, because people act on the number.

    This is why data engineers write checks that test the data itself, and not only whether the pipeline ran.

    Your turn: Which step of the pipeline did not do its job?

  7. Step 7 of 7

    Your first sentence

    Time to speak a little SQL yourself. Its simplest sentence has a fixed word order:

    SELECT column FROM table

    In words: "show me this column, from this table."

    The table below is stored under the name orders. Its headings, order_id, customer and city, are its columns.

    Your turn: Build the sentence that shows the city column from the orders table.

Reading is the easy part.

Every step above has an exercise. Do it for real, free, no account needed.

Start the lesson