Free lesson
The command line for data
Look inside huge files without opening them, count, filter and group from the terminal, and pull errors out of a log.
In this lesson you will
- Move around folders in a terminal
- Peek at, count and filter files of any size
- Chain small commands into a pipeline with |
Words you'll meet
- Current directory
- The folder the terminal is working in right now.
- Pipe ( | )
- Sends one command's output into the next command.
- Redirect ( > )
- Saves output to a file instead of showing it.
The lesson, step by step
Each step teaches one idea, then hands you a real terminal to try it on. The exercises run in your browser.
Step 1 of 7
Where am I?
A terminal always works inside one folder, the current directory.
pwdprints where you are.lslists what is here.cd datamoves into the data folder;cd ..goes back up one level.
~is shorthand for your home folder.Your turn: Print where you are, list what is here, then move into the data folder and list it.
Step 2 of 7
Peek, don't open
Real data files can be gigabytes. Opening one in an editor can freeze your laptop. Data engineers peek instead:
head orders.csvshows the first 10 lines;head -n 3 orders.csvshows 3.tailshows the last lines.
The first line of a CSV file is usually the header: the column names.
Your turn: Show just the first 3 lines of orders.csv.
Step 3 of 7
Count the rows
wc -l orders.csvcounts the lines in a file.It is the quickest sanity check after any load: did the number of rows you expected actually arrive? Remember the header is a line too.
Your turn: Count the lines in orders.csv.
Step 4 of 7
Find the lines you need
grep Austin orders.csvprints only the lines that containAustin.It is the command line's version of
WHERE, and it works on files of any size, very fast. Like SQL values, it cares about capitals:austinwould find nothing. Add-ito ignore case.Your turn: Show only the orders from Austin.
Step 5 of 7
Pipes
The pipe,
|, sends one command's output straight into the next command.grep Austin orders.csv | wc -lfinds the Austin lines, then counts them, with no file in between. Small tools snapped together: that is the whole idea of the command line.Your turn: Count the Denver orders using grep and wc -l joined by a pipe.
Step 6 of 7
Group and count
Three more tools, and you have a GROUP BY for files:
cut -d, -f3keeps only the 3rd comma-separated column.sortputs equal values next to each other.uniq -ccounts each run of equal lines.
Your turn: Count orders per city: cut the city column out of orders.csv, sort it, then count with uniq -c.
Step 7 of 7
Challenge
Recap.
grepfinds lines,|chains commands, and now one more:>saves the output into a file instead of the screen (replacing whatever was there).>>adds to the end instead.The pipeline failed last night. The log is in
~/logs/pipeline.log, and the on-call engineer wants just the errors.Your turn: Save every ERROR line from ~/logs/pipeline.log into a new file, ~/errors.txt.
Reading is the easy part.
Every step above has an exercise. Do it for real, free, no account needed.
Start the lesson