Pro lesson Advanced
Parquet and columnar storage
Why analytics files are stored by column, and what it buys you.
10 min7 hands-on steps
In this lesson you will
- See why analytics files store data by column
- Write the same orders to CSV and to Parquet, and compare them
- Use row groups and their statistics to skip data
Words you'll meet
- Row-oriented
- Stores one whole row, then the next. Good for apps that read and write one order at a time.
- Columnar
- Stores one whole column, then the next. Good for analytics, which reads a few columns of many rows.
- Parquet
- The standard columnar file format for analytics. It keeps each column's type inside the file.
- Compression
- Storing data in fewer bytes. Columns with few distinct values shrink the most.
- Row group
- A chunk of rows inside a Parquet file, stored column by column.
- Statistics
- The smallest and largest value of each column in each row group, kept in the file.
The steps
Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.
- 1Rows or columns?
- 2Write it both ways
- 3Types get guessed
- 4Types travel with Parquet
- 5Inside the file
- 6Row groups and statistics
- 7Challenge
This lesson is part of LearnDE Pro
$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.