Pro lesson Intermediate
Orchestration
Chain jobs into a DAG, handle retries, and backfill history.
10 min8 hands-on steps
In this lesson you will
- Model a pipeline as a DAG of tasks and find what a failure blocks
- Use the logical date, and make tasks safe to retry
- Find the days that need a backfill, run it, and alert only on what needs a person
Words you'll meet
- DAG
- Directed acyclic graph: tasks joined by one-way arrows that never loop back.
- Orchestrator
- The program that runs a DAG: starts tasks in order, retries them and records every run.
- Upstream / downstream
- The tasks a task waits for, and the tasks that wait for it.
- Retry
- Running a failed task again automatically, a few minutes later.
- Schedule
- When a DAG runs, such as every night at 02:00.
- Logical date
- The slice of data a run is responsible for, not the moment it runs.
- Backfill
- Running a task again for past logical dates.
The steps
Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.
- 1A pipeline is a DAG
- 2Where it starts
- 3Who is blocked?
- 4The logical date
- 5Retries need idempotent tasks
- 6What needs a backfill?
- 7Backfill
- 8Challenge
This lesson is part of LearnDE Pro
$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.