Pro lesson Intermediate
Data quality tests
Write the checks that catch bad data before your users do.
10 min9 hands-on steps
In this lesson you will
- Write a test as a query that returns the bad rows
- Test keys, missing values, allowed values, relationships, freshness and row counts
- Decide which failures stop the pipeline and which only warn
Words you'll meet
- Test (assertion)
- A rule about your data, written as a query that returns the rows breaking it. No rows means it passes.
- HAVING
- Like WHERE, but for groups: keeps only the groups that pass a test.
- Freshness
- How recent the newest data is. Correct but old data is a failure too.
- Severity
- How serious a failed test is: a warning to look at, or an error that stops the pipeline.
The steps
Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.
- 1A test returns the bad rows
- 2Unique keys
- 3Accepted values
- 4Relationships
- 5Freshness
- 6Count what you loaded
- 7Warn or fail
- 8Stop the line
- 9Challenge
This lesson is part of LearnDE Pro
$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.