Reliable pipelines

Pro lesson Intermediate

Incremental loads

Load only what changed since last time, without losing the rows that arrive late.

10 min8 hands-on steps

In this lesson you will

  • Load only the rows that changed, using a watermark
  • Replace changed rows instead of duplicating them, with an upsert
  • Catch rows that arrive late with a lookback window

Words you'll meet

Full reload
Empty the target table and copy every row from the source again.
Incremental load
Copy only the rows that are new or changed since the last load.
Watermark
The newest updated_at already loaded: how far the data has come.
Upsert
Load rows so new keys are added and existing keys are replaced (update + insert).
Late-arriving data
Rows that show up after newer rows have already been loaded.
Lookback
A fixed time before the watermark where each load starts, so late rows still get picked up.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1Full or incremental?
  2. 2The watermark
  3. 3Only what changed
  4. 4Upsert
  5. 5Late-arriving data
  6. 6The lookback
  7. 7How far back?
  8. 8Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.