Pro lesson Expert
Batch and streaming
Process data every night or every second: windows, late events, duplicates, and when streaming is worth what it costs.
10 min8 hands-on steps
In this lesson you will
- Weigh batch against streaming for a real request
- Count events in time windows, by the right clock
- Handle late events with a watermark, and duplicates from at-least-once delivery
Words you'll meet
- Event
- A record that something happened at a certain time, like "Austin sold a latte at 08:00:40".
- Stream
- An endless log of events. New events are added at the end; old ones are never changed.
- Window
- A slice of time that events are grouped into, like 08:00 to 08:05.
- Event time
- When the event happened.
- Processing time
- When the pipeline received the event.
- Watermark
- A line in event time: "I believe I have seen everything before this". Windows close when it passes them.
- At-least-once
- A delivery promise: every event arrives, but some arrive more than once.
- Micro-batch
- A small batch run every minute or so over whatever arrived since the last run.
The steps
Each step teaches one idea, then has you try it hands-on in your browser, and checks your work.
- 1Every night, or every second?
- 2An endless log
- 3Tumbling windows
- 4Event time vs processing time
- 5Late events and the watermark
- 6Delivered twice
- 7Is streaming worth it?
- 8Challenge
This lesson is part of LearnDE Pro
$12 a month or $100 a year: every lesson, spaced-repetition flashcards, and Learn by teaching. Refund on request within 14 days.