Pro lesson Expert
Make it run itself
Wire the steps into a DAG with incremental loads, tests and retries.
25 min11 hands-on steps
In this lesson you will
- Run the pipeline as a DAG of idempotent tasks, night after night
- Load facts incrementally with a watermark, and catch a late file with a lookback
- Hold the dashboard when a test fails, fix the cause, retry safely, and backfill one day
Words you'll meet
- View
- A saved query with a name. Reading it runs the query fresh, so it always shows the latest data.
- Held task
- A task the orchestrator skips because something upstream of it failed.
The steps
Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.
- 1The nightly DAG
- 2A new day lands
- 3The watermark
- 4Load only what is new
- 5Something is missing
- 6Why the watermark missed it
- 7Look back
- 8Tonight's run fails
- 9What is held
- 10Fix it and retry
- 11Challenge
This lesson is part of LearnDE Pro
$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.