Pro lesson Intermediate
Incremental loads
Load only what changed since last time, without losing the rows that arrive late.
10 min8 hands-on steps
In this lesson you will
- Load only the rows that changed, using a watermark
- Replace changed rows instead of duplicating them, with an upsert
- Catch rows that arrive late with a lookback window
Words you'll meet
- Full reload
- Empty the target table and copy every row from the source again.
- Incremental load
- Copy only the rows that are new or changed since the last load.
- Watermark
- The newest updated_at already loaded: how far the data has come.
- Upsert
- Load rows so new keys are added and existing keys are replaced (update + insert).
- Late-arriving data
- Rows that show up after newer rows have already been loaded.
- Lookback
- A fixed time before the watermark where each load starts, so late rows still get picked up.
The steps
Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.
- 1Full or incremental?
- 2The watermark
- 3Only what changed
- 4Upsert
- 5Late-arriving data
- 6The lookback
- 7How far back?
- 8Challenge
This lesson is part of LearnDE Pro
$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.