Pro lesson Advanced
Partitioning
Split big data so queries only read what they need.
10 min8 hands-on steps
In this lesson you will
- Split a table into folders by a column with COPY and PARTITION_BY
- Read the folders back, and see a filter skip whole folders
- Choose a partition key, and avoid the small files problem
- Load one day at a time into its own folder
Words you'll meet
- Partition
- One folder of a partitioned dataset, holding the rows that share one value of the key.
- Partition key
- The column whose value decides which folder a row goes into.
- Hive partitioning
- Naming each folder column=value, like city=Austin, so readers can tell what is inside without opening it.
- Partition pruning
- Skipping whole folders that a filter on the partition key rules out.
- Small files
- Too many tiny files: each one costs overhead and a separate read, so queries slow down.
The steps
Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.
- 1One file, every query
- 2Write the folders
- 3Read the folders back
- 4Skip whole folders
- 5Choose the key
- 6Too many small files
- 7Load one day
- 8Challenge
This lesson is part of LearnDE Pro
$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.