Free lesson
Why data engineering exists
No experience needed. Feel the problem first, then meet the tool that solves it.
In this lesson you will
- Why companies need data engineers
- What a table and a pipeline are
- Write your first line of SQL
Words you'll meet
- Table
- Data arranged in rows and columns, one fact per box.
- Pipeline
- Software that moves data from where it is made to where it is useful, with nobody touching it.
- SQL
- The language used to ask questions of tables.
The lesson, step by step
Each step teaches one idea, then hands you a real database to try it on. The exercises run in your browser.
Step 1 of 7
A simple question
Every business runs on records of what happened: sales, clicks, deliveries, payments. That collection of records is its data.
A small coffee chain writes down every order. Here are this week's. Each line is one order: where it was sold and how many drinks.
Your turn: Which city sold the most drinks? Click any order from that city.
Step 2 of 7
The same question, bigger
Data grows much faster than anyone can read it. A busy shop makes thousands of records a week; a website makes millions a day.
Business is good. The list now has 8,000 orders, and the owner asks you the same thing.
Your turn: Which city sold the most drinks?
Step 3 of 7
Ask the computer
A computer can do it in a blink, if you ask in a language it understands. The most common one for questions about data is called SQL.
The same question is written in SQL below. You do not need to understand it yet. Soon you will write ones like it yourself.
SQL has been around since the 1970s and almost every database speaks it. People say it as "sequel" or spell it out; both are fine.
The query here reads nearly like English: take the city and the total drinks, from the big table, one line per city, biggest first. It is the one skill every data engineering job asks for.
Your turn: Press Run.
Step 4 of 7
Do it by hand
That only worked because the orders were already in a neat grid, one fact per box. A grid like that is called a table.
Orders do not start out that way. They start as scribbles. Somebody has to file each one into the table. For the next minute, that somebody is you.
Each column holds one kind of fact and each row holds one order. Once data is in that shape, a computer can count it, add it up and sort it.
Turning scribbles into tables is called ingestion, and it is the first stage of nearly every pipeline. The scribbles might be text, web forms, clicks in an app or readings from a sensor.
Your turn: File each order slip into the table, one part at a time.
Step 5 of 7
Connect the steps
A pipeline is a row of steps. Data goes in one end and something useful comes out the other, with nobody touching it in between.
Real pipelines have the same stages under grander names: sources, transformation, storage (often called a data warehouse) and reporting. You will also hear ETL, short for extract, transform, load.
Most pipelines run on a schedule, such as every night. Some run all the time, handling each order the moment it happens.
Your turn: Click the four steps in the order the data should flow.
Step 6 of 7
When a step fails
The next morning the owner looks at the chart and frowns. Austin has two bars.
Pipelines rarely fail loudly. More often they keep running and produce a wrong number. That is worse than a crash, because people act on the number.
This is why data engineers write checks that test the data itself, and not only whether the pipeline ran.
Your turn: Which step of the pipeline did not do its job?
Step 7 of 7
Your first sentence
Time to speak a little SQL yourself. Its simplest sentence has a fixed word order:
SELECT column FROM table
In words: "show me this column, from this table."
The table below is stored under the name
orders. Its headings,order_id,customerandcity, are its columns.Your turn: Build the sentence that shows the city column from the orders table.
Reading is the easy part.
Every step above has an exercise. Do it for real, free, no account needed.
Start the lesson