Files, formats & storage

Pro lesson Advanced

Parquet and columnar storage

Why analytics files are stored by column, and what it buys you.

10 min7 hands-on steps

In this lesson you will

  • See why analytics files store data by column
  • Write the same orders to CSV and to Parquet, and compare them
  • Use row groups and their statistics to skip data

Words you'll meet

Row-oriented
Stores one whole row, then the next. Good for apps that read and write one order at a time.
Columnar
Stores one whole column, then the next. Good for analytics, which reads a few columns of many rows.
Parquet
The standard columnar file format for analytics. It keeps each column's type inside the file.
Compression
Storing data in fewer bytes. Columns with few distinct values shrink the most.
Row group
A chunk of rows inside a Parquet file, stored column by column.
Statistics
The smallest and largest value of each column in each row group, kept in the file.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1Rows or columns?
  2. 2Write it both ways
  3. 3Types get guessed
  4. 4Types travel with Parquet
  5. 5Inside the file
  6. 6Row groups and statistics
  7. 7Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.