Free lesson

Python gotchas that quietly corrupt data

Six Python habits that run without an error and still get the data wrong: float money, shared default lists, naive times, text encodings, ZIP codes and bare except.

15 min9 steps

In this lesson you will

  • Add up money exactly with Decimal
  • Spot bugs that raise no error: a shared default list, times without a time zone, an except that hides everything
  • Read files and ID codes without quietly changing them

Words you'll meet

Float
Python's everyday number with a decimal point. Fast, but stored in binary, so most cents are slightly off.
Decimal
A number type that stores decimal digits exactly. The right choice for money.
Default argument
The value a function uses for a parameter when the caller leaves it out.
Naive datetime
A date and time with no time zone attached, so it does not say which moment it means.
Encoding
The rule that turns letters into bytes in a file, and back. UTF-8 is the most common.
Exception
Python's way of reporting an error. try and except catch one so the program can carry on.
Start the lesson

The lesson, step by step

Each step teaches one idea, then hands you a real Python editor to try it on. The exercises run in your browser.

  1. Step 1 of 9

    Money is not a float

    The editor below runs real Python, right in your browser. Press Run (or Ctrl+Enter) to try your code.

    A float is Python's everyday number with a decimal point. Floats are stored in binary, and most amounts in cents, like 0.10, have no exact binary form. So Python stores something very close instead:

    print(0.1 + 0.2)   # 0.30000000000000004

    The error is tiny, but it shows up in totals, and it makes == fail on numbers that look equal. For money, use Decimal, which stores the digits exactly:

    from decimal import Decimal
    Decimal("4.50") + Decimal("0.60")   # Decimal('5.10')

    Build a Decimal from text, not from a float. Decimal(0.1) copies the float's error exactly: Decimal('0.1000000000000000055511151231257827021181583404541015625'). Prices usually arrive as text from a file anyway, so pass that text straight in.

    Databases have the same choice: a DECIMAL(10, 2) column is exact, a DOUBLE is a float.

    Your turn: Run the code to see the float total. Then change it so total is exact: make each price a Decimal from its text, and print the total.

  2. Step 2 of 9

    The default list everyone shares

    A default argument is the value a function uses when the caller leaves one out. Python makes the default once, when the def line runs. It does not make a new one for each call.

    For a number or a piece of text that is harmless. For a list or a dictionary, every call that relies on the default gets the same list, and anything one call adds is still there for the next.

    The fix is a default of None, and a fresh list made inside the function:

    def f(items=None):
        if items is None:
            items = []

    Your turn: Run the code: the second customer gets the first customer's oat milk. Fix add_topping so each order starts with its own empty list. We will call it for two more orders to check.

  3. Step 3 of 9

    Whose 8 a.m.?

    A datetime without a time zone is called naive. It says "8:00 on January 15", but not where, so it does not pin down one moment.

    The coffee chain's registers write local clock time. In January, Austin is 6 hours behind UTC (the world's reference clock) and Boston is 5 hours behind.

    • Austin's register logged an order at 8:00.
    • Boston's register logged an order at 8:30.

    Your turn: Which order really happened first?

  4. Step 4 of 9

    Attach the time zone

    To make a naive time aware, attach the time zone its clock was in, then convert to UTC so every store is on one clock:

    local = naive.replace(tzinfo=central)     # "this was Central time"
    utc = local.astimezone(timezone.utc)       # the same moment in UTC

    replace only labels the time; it does not move it. astimezone converts it. Store and compare times in UTC, and turn them into local time only to show people.

    Here the zones are fixed offsets, timezone(timedelta(hours=-6)). Real code should use a named zone such as ZoneInfo("America/Chicago") from the zoneinfo module, because offsets change with daylight saving time and a named zone knows when.

    Python also refuses to compare a naive time with an aware one: you get TypeError: can't compare offset-naive and offset-aware datetimes. That error is a gift. It tells you a time is missing its zone.

    Your turn: Set austin_utc and boston_utc to each register's time in UTC: attach its zone with replace(tzinfo=...), then convert with astimezone(timezone.utc).

  5. Step 5 of 9

    Text has an encoding

    A file holds bytes, not letters. An encoding is the rule that turns letters into bytes and back. UTF-8 is the most common one today, but plenty of files use others. Excel on Windows often saves a CSV as cp1252 (Windows-1252), where é is the single byte E9.

    open() without an encoding uses the computer's default, so the same code can work on one machine and fail on another. Here the default is UTF-8, and in UTF-8 that byte on its own is not allowed.

    Say which encoding the file uses: open("menu.csv", encoding="cp1252").

    A failure is the good outcome. The bad one is a wrong encoding that happens to work: UTF-8 bytes read as cp1252 come out as Café instead of Café, with no error, and that garbled text flows into your tables.

    A file does not say which encoding it uses. Ask whoever sends it, write the answer in the code, and when you write files yourself, write UTF-8.

    Your turn: menu.csv came from the Austin store's Windows PC. Run the code and read the error, then open the file with the cp1252 encoding so it prints correctly.

  6. Step 6 of 9

    pandas eats the zero

    Boston ZIP codes start with a zero: 02134. Here is the stores file, and how many pipelines read it:

    city,zip
    Boston,02134
    Austin,78701
    import pandas as pd
    stores = pd.read_csv("stores.csv")

    pandas, the most popular Python library for tables, guesses each column's type. The zip column holds only digits, so it guesses numbers, and the number 02134 is just 2134. The zero is gone, with no warning.

    ZIP codes, phone numbers, account numbers and product codes are made of digits, but they are not numbers: nobody adds two ZIP codes together. Keep them as text from the moment you read them.

    Spreadsheets do the same thing. Open the file in Excel, save it, and the zeros are gone for good.

    Your turn: Which line reads the file and keeps the zeros?

  7. Step 7 of 9

    Put the zeros back

    Sometimes the damage is already done: a file arrives with the zeros missing. Because every US ZIP code has exactly 5 digits, you can repair it. Turn the number into text, then pad it on the left with zeros:

    str(2134).zfill(5)   # '02134'

    This works only because the length is fixed. A phone number or account number that lost its zeros, with no fixed length, cannot be repaired this way. So read IDs as text in the first place.

    Your turn: Make fixed a list of 5-character ZIP code strings, with the zeros back.

  8. Step 8 of 9

    A bare except hides bugs

    An exception is how Python reports an error. try runs some code, and except catches the error so the program can carry on.

    A bare except:, with nothing after it, catches every error: the bad data you expected, but also your own typos, and even Ctrl+C. Name the error you expect instead. int("two") raises a ValueError, so:

    try:
        n = int(text)
    except ValueError:
        ...   # only bad numbers land here

    Your turn: This loop says every row is bad. Change except: to except ValueError:, run it to find the real bug, then fix that too.

  9. Step 9 of 9

    Challenge

    Recap. Money is Decimal, made from text. A default list is shared, so default to None. Times need a zone. Files need encoding=. IDs are text, and zfill repairs a fixed-length one. Catch only the error you expect.

    This function adds up each store's sales from orders.csv, the Windows export, and counts the rows it skips. It has five of today's gotchas. One more fact: Decimal("N/A") raises InvalidOperation from the decimal module (not ValueError).

    Your turn: Fix summarize: open the file as cp1252, add prices exactly with Decimal, catch only InvalidOperation, use 5-digit ZIP codes, and make each call start fresh. We will call it twice to check.

Reading is the easy part.

Every step above has an exercise. Do it for real, free, no account needed.

Start the lesson