Reliable pipelines

Pro lesson Intermediate

Orchestration

Chain jobs into a DAG, handle retries, and backfill history.

10 min8 hands-on steps

In this lesson you will

  • Model a pipeline as a DAG of tasks and find what a failure blocks
  • Use the logical date, and make tasks safe to retry
  • Find the days that need a backfill, run it, and alert only on what needs a person

Words you'll meet

DAG
Directed acyclic graph: tasks joined by one-way arrows that never loop back.
Orchestrator
The program that runs a DAG: starts tasks in order, retries them and records every run.
Upstream / downstream
The tasks a task waits for, and the tasks that wait for it.
Retry
Running a failed task again automatically, a few minutes later.
Schedule
When a DAG runs, such as every night at 02:00.
Logical date
The slice of data a run is responsible for, not the moment it runs.
Backfill
Running a task again for past logical dates.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1A pipeline is a DAG
  2. 2Where it starts
  3. 3Who is blocked?
  4. 4The logical date
  5. 5Retries need idempotent tasks
  6. 6What needs a backfill?
  7. 7Backfill
  8. 8Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.