Data Cleaning in Python: 10 Code-Along Examples

The data cleaning article makes the case that analysis is only as trustworthy as the data underneath it, and that most datasets arrive broken in predictable ways: wrong types, out-of-range values, duplicates, inconsistent categories, and gaps. This workbook fixes each of those in turn, on a small bike-rental dataset, and follows the article’s most important habit throughout: after every cleaning step, an assert that fails loudly if the fix did not hold. Clean in the right order, prove each step, and the data that reaches your analysis is data you can defend.

1. Inspect and convert types

The first cleaning step is always to check what pandas thinks each column is. info() reveals types, and astype corrects them, with category a memory-efficient type for columns holding a small set of repeated labels.

import pandas as pd
rentals = pd.DataFrame({
    "rental_id": [1, 2, 3, 4],
    "membership": ["basic", "premium", "basic", "premium"],
    "rating": ["5", "3", "4", "5"],      # numbers stored as text!
})
rentals.info()                            # spot the wrong types
# text-that-is-really-a-number -> integer
rentals["rating"] = rentals["rating"].astype("int")
# repeated labels -> category (smaller, faster)
rentals["membership"] = rentals["membership"].astype("category")
print(rentals.dtypes)
assert rentals["rating"].dtype == "int64"          # prove the fix
assert rentals["membership"].dtype == "category"

info() is the diagnosis: it shows rating stored as an object (text) when it should be numeric, which would make rentals["rating"].mean() fail or concatenate strings. astype("int") converts it, and astype("category") turns the low-cardinality membership column into pandas’ category type, which stores each label once and is faster to group on. The two assert lines are the habit to build: they turn a silent assumption into a checked fact, so if a later edit breaks the type, the failure surfaces here rather than deep in the analysis.

2. Clean strings before converting

Numbers often arrive wrapped in text: units, symbols, stray spaces. You must strip the text away before to_numeric will accept the value, and errors="coerce" turns anything still unparseable into NaN instead of crashing.

import pandas as pd
rentals = pd.DataFrame({
    "trip_duration": ["23 minutes", "45 minutes", "12 minutes", "unknown"],
})
cleaned = (rentals["trip_duration"]
           .str.replace(" minutes", "", regex=False)   # strip the unit
           .str.strip())                                # strip stray spaces
rentals["duration_min"] = pd.to_numeric(cleaned, errors="coerce")  # junk -> NaN
print(rentals)
print(rentals["duration_min"].isna().sum(), "value(s) could not be parsed")
assert rentals["duration_min"].dtype == "float64"

The chain removes the obstacles one at a time: .str.replace deletes the ” minutes” unit, and .str.strip removes surrounding whitespace, leaving a bare number in text form. pd.to_numeric then converts it, and errors="coerce" is the critical setting, because “unknown” cannot become a number and would otherwise raise; instead it becomes NaN, a missing value you can count and handle later with the tools in Example 9. Messy strings are just numbers wearing costumes, and this strip-then-coerce pattern undresses them safely.

3. Cap out-of-range values

Some values are simply impossible: a negative age, a rating above the maximum, a wheel larger than any real wheel. Masked assignment caps them at a sensible bound without dropping the row.

import pandas as pd
rentals = pd.DataFrame({
    "rental_id": [1, 2, 3, 4],
    "wheel_size": [26, 27, 40, 29],        # 40 is impossible
    "rating": [5, 3, 8, 4],                # 8 exceeds the 5-point scale
})
# cap wheel size at 27 inches
rentals.loc[rentals["wheel_size"] > 27, "wheel_size"] = 27
# cap rating at 5
rentals.loc[rentals["rating"] > 5, "rating"] = 5
print(rentals)
assert rentals["wheel_size"].max() <= 27
assert rentals["rating"].max() <= 5

The pattern df.loc[mask, "col"] = value is the workhorse of range cleaning: the mask rentals["wheel_size"] > 27 selects only the offending rows, and the assignment sets just their wheel_size to the cap, leaving every valid row untouched. Capping rather than deleting keeps the rest of the row’s information, which matters when only one field is wrong. The choice between capping and dropping is a judgement call, but the assert on the maximum proves whichever you chose actually held.

4. Parse dates and extract parts

Dates arrive as text and must be converted to real datetimes before you can compare, sort, or extract from them. pd.to_datetime does the conversion, and the .dt accessor pulls out components like month or day of week.

import pandas as pd
rentals = pd.DataFrame({
    "rental_id": [1, 2, 3],
    "rental_date": ["2026-01-15", "2026/02/03", "not a date"],
})
rentals["rental_date"] = pd.to_datetime(rentals["rental_date"], errors="coerce")
# now real date components are available
rentals["month"] = rentals["rental_date"].dt.month
rentals["weekday"] = rentals["rental_date"].dt.day_name()
print(rentals)
assert pd.api.types.is_datetime64_any_dtype(rentals["rental_date"])

pd.to_datetime parses varied formats, handling both the dash and slash separators here, and errors="coerce" again converts the unparseable “not a date” into NaT, the datetime version of NaN, rather than raising. Once the column is a real datetime, the .dt accessor unlocks everything time-related: .dt.month for the month number, .dt.day_name() for the weekday, and many more. Text dates are inert; converted dates are queryable, which is why parsing is a prerequisite for any time-based analysis.

5. Handle duplicates, exact and partial

Duplicates come in two forms. Exact duplicates are identical rows and drop_duplicates removes them. Partial duplicates share a key but disagree on some values, and you resolve those by grouping and aggregating.

import pandas as pd
rentals = pd.DataFrame({
    "rental_id": [1, 1, 2, 3, 3],
    "station": ["North", "North", "South", "East", "East"],
    "duration": [23, 23, 45, 12, 18],     # id 3 disagrees: 12 vs 18
})
# inspect: keep=False flags ALL copies, not just the extras
print("all duplicated rows:")
print(rentals[rentals.duplicated(subset="rental_id", keep=False)])
# exact duplicates (id 1) vanish; partial (id 3) need a decision
deduped = rentals.drop_duplicates(subset=["rental_id", "station", "duration"])
# resolve partial duplicates by aggregating
resolved = (rentals
            .groupby(["rental_id", "station"], as_index=False)
            .agg(duration=("duration", "mean")))
print("\nresolved by averaging:")
print(resolved)
assert resolved["rental_id"].is_unique

The keep=False argument is the inspection tool: it flags every copy of a duplicated key rather than just the surplus ones, so you can see the full picture before deciding. Rental 1’s rows are identical, so drop_duplicates collapses them safely. Rental 3 is the harder case, two rows with the same id but different durations, which no automatic dedup can resolve because it does not know which is right; the fix is to groupby the key and agg a decision, here averaging the durations. The final asserton uniqueness proves one row per rental remains.

6. Validate categories against an allowed set

A category column should only contain known values. Set algebra finds the intruders: subtract the allowed set from the values present, and whatever remains is invalid.

import pandas as pd
rentals = pd.DataFrame({
    "rental_id": [1, 2, 3, 4, 5],
    "bike_type": ["road", "mountain", "hybrid", "unicycle", "road"],
})
allowed = {"road", "mountain", "hybrid", "electric"}
# which values appear that should not?
present = set(rentals["bike_type"])
invalid = present - allowed
print("invalid categories:", invalid)         # {'unicycle'}
# keep only the rows whose category is allowed
clean = rentals[rentals["bike_type"].isin(allowed)]
print(clean)
assert set(clean["bike_type"]).issubset(allowed)

Set difference is the clean way to find category errors: present - allowed returns exactly the values that appear in the data but are not on the approved list, here the stray “unicycle”. Once you know the offenders, .isin(allowed) builds a boolean mask of the valid rows, and indexing with it drops the rest, though you might instead correct them if they are fixable typos, which is Example 7’s job. The assert that the remaining categories are a subset of the allowed set proves no intruder survived.

7. Fix inconsistent categories and bin numbers

Categories are often technically valid but inconsistently written: different cases, stray spaces, or too many fine-grained labels. String cleaning normalises them, a replacement dictionary collapses them, and pd.cut bins a numeric column into labelled ranges.

import pandas as pd
rentals = pd.DataFrame({
    "cleanliness": [" Clean ", "clean", "DIRTY", "Dirty", "spotless"],
    "duration": [10, 35, 55, 80, 20],
})
# normalise: strip spaces and lowercase
rentals["cleanliness"] = rentals["cleanliness"].str.strip().str.lower()
# collapse variants into canonical labels
rentals["cleanliness"] = rentals["cleanliness"].replace({"spotless": "clean"})
# bin a numeric column into labelled ranges
rentals["length_band"] = pd.cut(
    rentals["duration"],
    bins=[0, 30, 60, 999],
    labels=["short", "medium", "long"],
)
print(rentals)
assert set(rentals["cleanliness"]) == {"clean", "dirty"}

The two steps solve two different inconsistencies. String cleaning fixes presentation: .str.strip().str.lower() turns ” Clean “, “clean”, and “DIRTY” into a consistent lowercase form, so what were four distinct-looking values become two. The replacement dictionary then fixes vocabulary, folding the synonym “spotless” into “clean”. Finally, pd.cut addresses the opposite problem of too much numeric detail, grouping continuous durations into three named bands, which is how you turn a messy numeric column into a tidy categorical one for reporting.

8. Cross-field validation

Some errors are only visible by comparing columns. If a row claims an age that does not match its birth year, one of the two is wrong. Cross-field checks catch these internal contradictions.

import pandas as pd
current_year = 2026
riders = pd.DataFrame({
    "rider_id": [1, 2, 3],
    "birth_year": [1990, 2000, 1985],
    "age": [36, 26, 30],            # rider 3: 2026-1985=41, not 30!
})
# the age implied by the birth year
riders["expected_age"] = current_year - riders["birth_year"]
# flag rows where the stated age disagrees
riders["age_ok"] = riders["age"] == riders["expected_age"]
print(riders)
inconsistent = riders[~riders["age_ok"]]
print("\ninconsistent rows:")
print(inconsistent)
assert riders.loc[riders["age_ok"], "age"].equals(
    riders.loc[riders["age_ok"], "expected_age"])

The check recomputes what one column should be from another, current_year - birth_year should equal the stated age, and flags any row where they disagree. Rider 3 is caught: born in 1985, they should be 41, not the 30 recorded. Cross-field validation is powerful precisely because neither column looks wrong on its own; 30 is a perfectly plausible age and 1985 a perfectly plausible year, and only their relationship exposes the error. Once flagged, you decide which field to trust, but the contradiction is now visible rather than lurking.

9. Handle missing data

Missing values are the most common flaw. First see how much is missing and whether it is random, then decide per column whether to drop the rows or fill the gaps with a sensible value.

import pandas as pd
import numpy as np
rentals = pd.DataFrame({
    "rental_id": [1, 2, 3, 4, 5],
    "duration": [23, np.nan, 45, np.nan, 30],
    "station": ["North", "South", np.nan, "East", "West"],
})
print(rentals.isna().sum())               # how much is missing, per column
# drop rows missing the essential key column
rentals = rentals.dropna(subset=["station"])
# fill numeric gaps with a computed value
median_duration = rentals["duration"].median()
rentals["duration"] = rentals["duration"].fillna(median_duration)
print(rentals)
assert rentals["duration"].isna().sum() == 0
assert rentals["station"].isna().sum() == 0

The first move is always to measure: isna().sum() counts the gaps per column so you know the scale of the problem before choosing a strategy. The strategy then varies by column: station is essential and hard to guess, so rows missing it are dropped with dropna(subset=...); duration is numeric and can be reasonably estimated, so its gaps are filled with the column median via fillna. The median is preferred over the mean here because it resists outliers, and the assert that no NaN remains proves the column is now complete. For seeing whether missingness is random or clustered, the missingno library draws a matrix, which is worth adding when the pattern matters.

10. Fuzzy matching for typos

The hardest inconsistencies are near-misses: “Mountain”, “Montain”, “Mountian” all mean the same category but no exact match unites them. Fuzzy matching scores string similarity and maps each variant to its closest canonical label.

import difflib
# the canonical categories you want everything to map to
canonical = ["road", "mountain", "hybrid", "electric"]
# messy values as they arrived, with typos
messy = ["Road", "montain", "mountian", "hibrid", "electirc"]
for value in messy:
    match = difflib.get_close_matches(
        value.lower(), canonical, n=1, cutoff=0.6   # similarity threshold
    )
    result = match[0] if match else "NO MATCH"
    print(f"{value:10s} -> {result}")
# prove every messy value found a canonical home
mapped = [difflib.get_close_matches(v.lower(), canonical, n=1, cutoff=0.6)
          for v in messy]
assert all(len(m) == 1 for m in mapped)

difflib.get_close_matches scores how similar each messy value is to every canonical label and returns the best one above the cutoff threshold, so “montain” and “mountian” both resolve to “mountain” despite never matching it exactly. The threshold is the dial: too low and unrelated words match, too high and real typos slip through, so the value is tuned to your data. In production the thefuzz library offers more similarity metrics and a common cutoff around 80 out of 100, and for deduplicating records across two whole tables the recordlinkage library adds blocking, which compares only plausible pairs to keep the work tractable. The principle is the same: when exact matching fails, similarity matching rescues the categories that a human would recognise at a glance.

Work through these and you have the whole article in practice: type inspection and conversion, string cleaning before numeric coercion, range capping, date parsing with the .dt accessor, exact and partial duplicate resolution, category validation by set algebra, inconsistency fixing and binning, cross-field validation, missing-data strategy, and fuzzy matching for typos. The discipline that ties it together is the assert after every step, which the article insists on for good reason: cleaning is a sequence of small promises about the data, and an assertion is how you keep each promise honest before the next step depends on it.

Hope these articles help.