Files, formats & storage

Pro lesson Advanced

Partitioning

Split big data so queries only read what they need.

10 min8 hands-on steps

In this lesson you will

  • Split a table into folders by a column with COPY and PARTITION_BY
  • Read the folders back, and see a filter skip whole folders
  • Choose a partition key, and avoid the small files problem
  • Load one day at a time into its own folder

Words you'll meet

Partition
One folder of a partitioned dataset, holding the rows that share one value of the key.
Partition key
The column whose value decides which folder a row goes into.
Hive partitioning
Naming each folder column=value, like city=Austin, so readers can tell what is inside without opening it.
Partition pruning
Skipping whole folders that a filter on the partition key rules out.
Small files
Too many tiny files: each one costs overhead and a separate read, so queries slow down.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1One file, every query
  2. 2Write the folders
  3. 3Read the folders back
  4. 4Skip whole folders
  5. 5Choose the key
  6. 6Too many small files
  7. 7Load one day
  8. 8Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.