Architecture at scale

Pro lesson Expert

Batch and streaming

Process data every night or every second: windows, late events, duplicates, and when streaming is worth what it costs.

10 min8 hands-on steps

In this lesson you will

  • Weigh batch against streaming for a real request
  • Count events in time windows, by the right clock
  • Handle late events with a watermark, and duplicates from at-least-once delivery

Words you'll meet

Event
A record that something happened at a certain time, like "Austin sold a latte at 08:00:40".
Stream
An endless log of events. New events are added at the end; old ones are never changed.
Window
A slice of time that events are grouped into, like 08:00 to 08:05.
Event time
When the event happened.
Processing time
When the pipeline received the event.
Watermark
A line in event time: "I believe I have seen everything before this". Windows close when it passes them.
At-least-once
A delivery promise: every event arrives, but some arrive more than once.
Micro-batch
A small batch run every minute or so over whatever arrived since the last run.

The steps

Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.

  1. 1Every night, or every second?
  2. 2An endless log
  3. 3Tumbling windows
  4. 4Event time vs processing time
  5. 5Late events and the watermark
  6. 6Delivered twice
  7. 7Is streaming worth it?
  8. 8Challenge

This lesson is part of LearnDE Pro

$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.