The tools nobody teaches

Pro lesson All levels

When a pipeline fails at 2am

An incident drill: read the logs, find the first error and the row behind it, set the bad rows aside, and plan the backfill.

15 min9 hands-on steps

In this lesson you will

  • Read a pipeline log and find the first error, not the loudest
  • Trace an error back to the row that caused it
  • Set bad rows aside, plan a backfill, and write up what happened

Words you'll meet

On call
Taking your turn to be woken up when a pipeline breaks.
Upstream / downstream
The tasks a task depends on, and the tasks that depend on it.
Quarantine
Setting bad rows aside, untouched, so the good ones can load and nothing is lost.
Backfill
Re-running a pipeline for days it missed.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1Start at the end
  2. 2How long has it been broken?
  3. 3The first error, not the loudest
  4. 4Upstream and downstream
  5. 5Find the bad row
  6. 6How far did it spread?
  7. 7Quarantine, do not delete
  8. 8Plan the backfill
  9. 9Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.