Reliable pipelines

Pro lesson Intermediate

Data quality tests

Write the checks that catch bad data before your users do.

10 min9 hands-on steps

In this lesson you will

  • Write a test as a query that returns the bad rows
  • Test keys, missing values, allowed values, relationships, freshness and row counts
  • Decide which failures stop the pipeline and which only warn

Words you'll meet

Test (assertion)
A rule about your data, written as a query that returns the rows breaking it. No rows means it passes.
HAVING
Like WHERE, but for groups: keeps only the groups that pass a test.
Freshness
How recent the newest data is. Correct but old data is a failure too.
Severity
How serious a failed test is: a warning to look at, or an error that stops the pipeline.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1A test returns the bad rows
  2. 2Unique keys
  3. 3Accepted values
  4. 4Relationships
  5. 5Freshness
  6. 6Count what you loaded
  7. 7Warn or fail
  8. 8Stop the line
  9. 9Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.