Capstone: your own pipeline

Pro lesson Expert

Make it run itself

Wire the steps into a DAG with incremental loads, tests and retries.

25 min11 hands-on steps

In this lesson you will

  • Run the pipeline as a DAG of idempotent tasks, night after night
  • Load facts incrementally with a watermark, and catch a late file with a lookback
  • Hold the dashboard when a test fails, fix the cause, retry safely, and backfill one day

Words you'll meet

View
A saved query with a name. Reading it runs the query fresh, so it always shows the latest data.
Held task
A task the orchestrator skips because something upstream of it failed.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1The nightly DAG
  2. 2A new day lands
  3. 3The watermark
  4. 4Load only what is new
  5. 5Something is missing
  6. 6Why the watermark missed it
  7. 7Look back
  8. 8Tonight's run fails
  9. 9What is held
  10. 10Fix it and retry
  11. 11Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.