Available now

When a pipeline fails at 2am.

Every pipeline fails eventually. What matters is how fast you find out why.

This lesson is an incident drill. The nightly job has failed and the dashboard is stale: read the logs, find the row behind the error, set the bad data aside, and plan the reload of the days you missed.

Why data engineers need it

  • Being on call is part of most data engineering jobs.
  • Bad data rarely raises an error. Logs and checks are how you notice.
  • Fixing and backfilling safely is what separates seniors from juniors.

What you will learn

  • Reading a pipeline log, starting from the end
  • Finding the first error, not the loudest one
  • Upstream and downstream: which task to fix
  • Tracing an error back to the row that caused it
  • Quarantining bad rows instead of deleting them
  • Planning a backfill and writing up what happened

The lessons

  1. 1When a pipeline fails at 2amProAn incident drill: read the logs, find the first error and the row behind it, set the bad rows aside, and plan the backfill.

Nothing to install. Nothing to break.

Every lesson runs in your browser, on your laptop or your phone.

Try it now