Free lesson
Python gotchas that quietly corrupt data
Six Python habits that run without an error and still get the data wrong: float money, shared default lists, naive times, text encodings, ZIP codes and bare except.
In this lesson you will
- Add up money exactly with Decimal
- Spot bugs that raise no error: a shared default list, times without a time zone, an except that hides everything
- Read files and ID codes without quietly changing them
Words you'll meet
- Float
- Python's everyday number with a decimal point. Fast, but stored in binary, so most cents are slightly off.
- Decimal
- A number type that stores decimal digits exactly. The right choice for money.
- Default argument
- The value a function uses for a parameter when the caller leaves it out.
- Naive datetime
- A date and time with no time zone attached, so it does not say which moment it means.
- Encoding
- The rule that turns letters into bytes in a file, and back. UTF-8 is the most common.
- Exception
- Python's way of reporting an error. try and except catch one so the program can carry on.
The lesson, step by step
Each step teaches one idea, then hands you a real Python editor to try it on. The exercises run in your browser.
Step 1 of 9
Money is not a float
The editor below runs real Python, right in your browser. Press Run (or Ctrl+Enter) to try your code.
A float is Python's everyday number with a decimal point. Floats are stored in binary, and most amounts in cents, like 0.10, have no exact binary form. So Python stores something very close instead:
print(0.1 + 0.2) # 0.30000000000000004
The error is tiny, but it shows up in totals, and it makes
==fail on numbers that look equal. For money, use Decimal, which stores the digits exactly:from decimal import Decimal Decimal("4.50") + Decimal("0.60") # Decimal('5.10')Build a Decimal from text, not from a float.
Decimal(0.1)copies the float's error exactly:Decimal('0.1000000000000000055511151231257827021181583404541015625'). Prices usually arrive as text from a file anyway, so pass that text straight in.Databases have the same choice: a
DECIMAL(10, 2)column is exact, aDOUBLEis a float.Your turn: Run the code to see the float total. Then change it so total is exact: make each price a Decimal from its text, and print the total.
Step 2 of 9
The default list everyone shares
A default argument is the value a function uses when the caller leaves one out. Python makes the default once, when the
defline runs. It does not make a new one for each call.For a number or a piece of text that is harmless. For a list or a dictionary, every call that relies on the default gets the same list, and anything one call adds is still there for the next.
The fix is a default of
None, and a fresh list made inside the function:def f(items=None): if items is None: items = []Your turn: Run the code: the second customer gets the first customer's oat milk. Fix add_topping so each order starts with its own empty list. We will call it for two more orders to check.
Step 3 of 9
Whose 8 a.m.?
A
datetimewithout a time zone is called naive. It says "8:00 on January 15", but not where, so it does not pin down one moment.The coffee chain's registers write local clock time. In January, Austin is 6 hours behind UTC (the world's reference clock) and Boston is 5 hours behind.
- Austin's register logged an order at 8:00.
- Boston's register logged an order at 8:30.
Your turn: Which order really happened first?
Step 4 of 9
Attach the time zone
To make a naive time aware, attach the time zone its clock was in, then convert to UTC so every store is on one clock:
local = naive.replace(tzinfo=central) # "this was Central time" utc = local.astimezone(timezone.utc) # the same moment in UTC
replaceonly labels the time; it does not move it.astimezoneconverts it. Store and compare times in UTC, and turn them into local time only to show people.Here the zones are fixed offsets,
timezone(timedelta(hours=-6)). Real code should use a named zone such asZoneInfo("America/Chicago")from thezoneinfomodule, because offsets change with daylight saving time and a named zone knows when.Python also refuses to compare a naive time with an aware one: you get
TypeError: can't compare offset-naive and offset-aware datetimes. That error is a gift. It tells you a time is missing its zone.Your turn: Set austin_utc and boston_utc to each register's time in UTC: attach its zone with replace(tzinfo=...), then convert with astimezone(timezone.utc).
Step 5 of 9
Text has an encoding
A file holds bytes, not letters. An encoding is the rule that turns letters into bytes and back. UTF-8 is the most common one today, but plenty of files use others. Excel on Windows often saves a CSV as cp1252 (Windows-1252), where é is the single byte
E9.open()without an encoding uses the computer's default, so the same code can work on one machine and fail on another. Here the default is UTF-8, and in UTF-8 that byte on its own is not allowed.Say which encoding the file uses:
open("menu.csv", encoding="cp1252").A failure is the good outcome. The bad one is a wrong encoding that happens to work: UTF-8 bytes read as cp1252 come out as
Caféinstead ofCafé, with no error, and that garbled text flows into your tables.A file does not say which encoding it uses. Ask whoever sends it, write the answer in the code, and when you write files yourself, write UTF-8.
Your turn: menu.csv came from the Austin store's Windows PC. Run the code and read the error, then open the file with the cp1252 encoding so it prints correctly.
Step 6 of 9
pandas eats the zero
Boston ZIP codes start with a zero:
02134. Here is the stores file, and how many pipelines read it:city,zip Boston,02134 Austin,78701
import pandas as pd stores = pd.read_csv("stores.csv")pandas, the most popular Python library for tables, guesses each column's type. The zip column holds only digits, so it guesses numbers, and the number 02134 is just 2134. The zero is gone, with no warning.
ZIP codes, phone numbers, account numbers and product codes are made of digits, but they are not numbers: nobody adds two ZIP codes together. Keep them as text from the moment you read them.
Spreadsheets do the same thing. Open the file in Excel, save it, and the zeros are gone for good.
Your turn: Which line reads the file and keeps the zeros?
Step 7 of 9
Put the zeros back
Sometimes the damage is already done: a file arrives with the zeros missing. Because every US ZIP code has exactly 5 digits, you can repair it. Turn the number into text, then pad it on the left with zeros:
str(2134).zfill(5) # '02134'
This works only because the length is fixed. A phone number or account number that lost its zeros, with no fixed length, cannot be repaired this way. So read IDs as text in the first place.
Your turn: Make fixed a list of 5-character ZIP code strings, with the zeros back.
Step 8 of 9
A bare except hides bugs
An exception is how Python reports an error.
tryruns some code, andexceptcatches the error so the program can carry on.A bare
except:, with nothing after it, catches every error: the bad data you expected, but also your own typos, and even Ctrl+C. Name the error you expect instead.int("two")raises aValueError, so:try: n = int(text) except ValueError: ... # only bad numbers land hereYour turn: This loop says every row is bad. Change except: to except ValueError:, run it to find the real bug, then fix that too.
Step 9 of 9
Challenge
Recap. Money is Decimal, made from text. A default list is shared, so default to
None. Times need a zone. Files needencoding=. IDs are text, andzfillrepairs a fixed-length one. Catch only the error you expect.This function adds up each store's sales from
orders.csv, the Windows export, and counts the rows it skips. It has five of today's gotchas. One more fact:Decimal("N/A")raisesInvalidOperationfrom thedecimalmodule (not ValueError).Your turn: Fix summarize: open the file as cp1252, add prices exactly with Decimal, catch only InvalidOperation, use 5-digit ZIP codes, and make each call start fresh. We will call it twice to check.
Reading is the easy part.
Every step above has an exercise. Do it for real, free, no account needed.
Start the lesson