Free lesson

The command line for data

Look inside huge files without opening them, count, filter and group from the terminal, and pull errors out of a log.

12 min7 steps

In this lesson you will

  • Move around folders in a terminal
  • Peek at, count and filter files of any size
  • Chain small commands into a pipeline with |

Words you'll meet

Current directory
The folder the terminal is working in right now.
Pipe ( | )
Sends one command's output into the next command.
Redirect ( > )
Saves output to a file instead of showing it.
Start the lesson

The lesson, step by step

Each step teaches one idea, then hands you a real terminal to try it on. The exercises run in your browser.

  1. Step 1 of 7

    Where am I?

    A terminal always works inside one folder, the current directory.

    • pwd prints where you are.
    • ls lists what is here.
    • cd data moves into the data folder; cd .. goes back up one level.

    ~ is shorthand for your home folder.

    Your turn: Print where you are, list what is here, then move into the data folder and list it.

  2. Step 2 of 7

    Peek, don't open

    Real data files can be gigabytes. Opening one in an editor can freeze your laptop. Data engineers peek instead:

    • head orders.csv shows the first 10 lines; head -n 3 orders.csv shows 3.
    • tail shows the last lines.

    The first line of a CSV file is usually the header: the column names.

    Your turn: Show just the first 3 lines of orders.csv.

  3. Step 3 of 7

    Count the rows

    wc -l orders.csv counts the lines in a file.

    It is the quickest sanity check after any load: did the number of rows you expected actually arrive? Remember the header is a line too.

    Your turn: Count the lines in orders.csv.

  4. Step 4 of 7

    Find the lines you need

    grep Austin orders.csv prints only the lines that contain Austin.

    It is the command line's version of WHERE, and it works on files of any size, very fast. Like SQL values, it cares about capitals: austin would find nothing. Add -i to ignore case.

    Your turn: Show only the orders from Austin.

  5. Step 5 of 7

    Pipes

    The pipe, |, sends one command's output straight into the next command.

    grep Austin orders.csv | wc -l finds the Austin lines, then counts them, with no file in between. Small tools snapped together: that is the whole idea of the command line.

    Your turn: Count the Denver orders using grep and wc -l joined by a pipe.

  6. Step 6 of 7

    Group and count

    Three more tools, and you have a GROUP BY for files:

    • cut -d, -f3 keeps only the 3rd comma-separated column.
    • sort puts equal values next to each other.
    • uniq -c counts each run of equal lines.

    Your turn: Count orders per city: cut the city column out of orders.csv, sort it, then count with uniq -c.

  7. Step 7 of 7

    Challenge

    Recap. grep finds lines, | chains commands, and now one more: > saves the output into a file instead of the screen (replacing whatever was there). >> adds to the end instead.

    The pipeline failed last night. The log is in ~/logs/pipeline.log, and the on-call engineer wants just the errors.

    Your turn: Save every ERROR line from ~/logs/pipeline.log into a new file, ~/errors.txt.

Reading is the easy part.

Every step above has an exercise. Do it for real, free, no account needed.

Start the lesson