Capstone: your own pipeline

Pro lesson Expert

Land the raw data

Copy the shops' messy till files into raw tables, untouched, with a load you can safely run twice.

20 min11 hands-on steps

In this lesson you will

  • Look at the source files before loading anything
  • Land every file into a raw table as text, with the file name and load time on each row
  • Make the load safe to retry, and reconcile it against what the tills say they sent

Words you'll meet

Landing folder
Where source files arrive before anything reads them.
Wildcard
A * in a file name that matches any text, so one pattern reads many files.
Lineage
Where each row came from: here, the file name and the time it was loaded.
Manifest
A list a source sends of the files it delivered, with a row count for each.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1The job
  2. 2Open a file first
  3. 3Let nothing be guessed
  4. 4Every file at once
  5. 5Land it
  6. 6The scheduler retried
  7. 7Safe to run twice
  8. 8Reconcile with the manifest
  9. 9Leave the mess in raw
  10. 10The customers export
  11. 11Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.