Pro lesson All levels
When a pipeline fails at 2am
An incident drill: read the logs, find the first error and the row behind it, set the bad rows aside, and plan the backfill.
15 min9 hands-on steps
In this lesson you will
- Read a pipeline log and find the first error, not the loudest
- Trace an error back to the row that caused it
- Set bad rows aside, plan a backfill, and write up what happened
Words you'll meet
- On call
- Taking your turn to be woken up when a pipeline breaks.
- Upstream / downstream
- The tasks a task depends on, and the tasks that depend on it.
- Quarantine
- Setting bad rows aside, untouched, so the good ones can load and nothing is lost.
- Backfill
- Re-running a pipeline for days it missed.
The steps
Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.
- 1Start at the end
- 2How long has it been broken?
- 3The first error, not the loudest
- 4Upstream and downstream
- 5Find the bad row
- 6How far did it spread?
- 7Quarantine, do not delete
- 8Plan the backfill
- 9Challenge
This lesson is part of LearnDE Pro
$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.