Categorical Data in Pandas: 10 Code-Along Examples

The categorical data article explains the core idea: a category column stores each distinct value once in a lookup table and replaces the actual data with small integer codes. That single trick delivers memory savings, meaningful ordering, and cleaner grouping and encoding for machine learning. This workbook runs the whole toolkit: creating categoricals three ways, the .cat accessor, ordered categories, managing the category set without silently creating missing values, the cleaning pipeline that should always come first, and the two encodings that feed models. Run them in order; the ideas stack.

1. The category dtype and the memory win

Start with the reason categoricals exist. A text column repeats the same strings over and over; a category column stores each unique value once and uses integer codes underneath. Measure the difference with .nbytes.

import pandas as pd
import numpy as np
rng = np.random.default_rng(0)
departments = rng.choice(["Sales", "Engineering", "Support", "Finance"], size=10_000)
as_object = pd.Series(departments)                    # plain text column
as_category = pd.Series(departments, dtype="category")  # categorical column
print("object dtype:  ", as_object.nbytes, "bytes")
print("category dtype:", as_category.nbytes, "bytes")
print("dtype:", as_category.dtype)
# object dtype:   80000 bytes
# category dtype: 10132 bytes  (roughly 8x smaller)

Ten thousand rows holding only four distinct values is exactly the shape categoricals are built for: the four strings are stored once, and every row is just a one-byte code pointing at them. On a real dataset with millions of rows, this is the difference between a DataFrame that fits in memory and one that does not.

2. Three ways to create a categorical

You can convert an existing column, declare the type at load time so the memory saving applies from the first byte, or construct one directly with pd.Categorical, which is also where ordering enters.

import pandas as pd
from io import StringIO
# way 1: convert an existing column
staff = pd.DataFrame({"department": ["Sales", "Support", "Sales", "Finance"]})
staff["department"] = staff["department"].astype("category")
# way 2: declare the dtype at CSV load time
csv = StringIO("name,department\nAva,Sales\nBen,Support\nChloe,Sales")
loaded = pd.read_csv(csv, dtype={"department": "category"})
# way 3: construct directly, with an explicit order
medals = pd.Categorical(
    ["Silver", "Gold", "Bronze", "Gold"],
    categories=["Bronze", "Silver", "Gold"],
    ordered=True
)
print(staff["department"].dtype)     # category
print(loaded["department"].dtype)    # category
print(medals)                        # ['Silver', 'Gold', 'Bronze', 'Gold']
                                     # Categories (3, object): ['Bronze' < 'Silver' < 'Gold']

The astype route is the everyday one, the dtype= argument at load time means a huge CSV never materialises as wasteful text in the first place, and pd.Categorical with ordered=True is how you tell pandas that Bronze, Silver, and Gold are not just labels but a ranking. The < signs in the printed categories are that ordering made visible.

3. Inside the column: categories and codes

The .cat accessor opens the machinery. .cat.categories is the lookup table and .cat.codes is the integer per row, with -1 reserved for missing values.

import pandas as pd
import numpy as np
sizes = pd.Series(["small", "large", "medium", np.nan, "small"], dtype="category")
print("categories:", list(sizes.cat.categories))
print("codes:     ", list(sizes.cat.codes))
# categories: ['large', 'medium', 'small']
# codes:      [2, 0, 1, -1, 2]

Each row’s code is its position in the categories list, so both “small” rows share the code 2, and the missing value gets -1 rather than a code of its own. Seeing this mapping once demystifies everything else: every categorical operation, from memory savings to label encoding, is a manipulation of exactly these two pieces.

4. Ordered categories: sorting and comparing that make sense

Once a categorical is ordered, sorting follows the ranking instead of the alphabet, and comparison operators work. This is what plain text columns can never give you.

import pandas as pd
survey = pd.DataFrame({"respondent": ["A", "B", "C", "D", "E"],
                       "education": ["Masters", "GCSE", "PhD", "Bachelors", "GCSE"]})
order = ["GCSE", "A-Level", "Bachelors", "Masters", "PhD"]
survey["education"] = pd.Categorical(survey["education"],
                                     categories=order, ordered=True)
print(survey.sort_values("education"))
# GCSE rows first, PhD last: ranked, not alphabetical
degree_or_higher = survey[survey["education"] >= "Bachelors"]
print(degree_or_higher["respondent"].tolist())   # ['A', 'C', 'D']

Alphabetical sorting would put Bachelors before GCSE, which is meaningless; the ordered categorical sorts by seniority of qualification instead. And the filter >= "Bachelors" reads exactly like the question it answers. Any scale with a natural order, satisfaction ratings, size bands, income brackets, deserves this treatment.

5. Managing the category set, and the NaN trap

The .cat accessor also edits the lookup table: add categories before assigning new values, rename them with a dictionary, and remove them, with one sharp edge to know about.

import pandas as pd
coats = pd.Series(["short", "long", "short", "wire"], dtype="category")
# add a category before assigning it, or pandas refuses the value
coats = coats.cat.add_categories(["hairless"])
# rename with a dictionary
coats = coats.cat.rename_categories({"wire": "wirehaired"})
print(list(coats.cat.categories))   # ['long', 'short', 'wirehaired', 'hairless']
# THE TRAP: removing a category converts its rows to NaN silently
dropped = coats.cat.remove_categories(["wirehaired"])
print(dropped.isna().sum())         # 1 row just became missing

The trap in the last step is the one the guide warns about: remove_categories does not delete rows or reassign them, it turns every row holding that category into NaN without a word. The safe pattern is to reassign those rows to another category first, and only then remove the now-unused label.

6. Clean first, convert second

Real categorical columns arrive messy: stray capitals, trailing spaces, typos. Convert those to category and every variant becomes its own category, so cleaning always comes first. The pipeline is replace, strip, lower, then convert.

import pandas as pd
raw = pd.Series(["Sales", "sales ", " SALES", "Suport", "support", "Finance"])
print("before:", raw.nunique(), "distinct values")   # 6, though only 3 are real
cleaned = (raw
           .str.strip()                          # kill stray whitespace
           .str.lower()                          # one casing
           .replace({"suport": "support"})       # fix the typo
           .astype("category"))                  # NOW convert
print("after: ", list(cleaned.cat.categories))   # ['finance', 'sales', 'support']

Six apparent departments collapse into the three that actually exist. Convert before cleaning and you would be managing six categories, three of them junk, through every operation that follows. Categorical hygiene is exactly this: make the strings canonical first, then lock them in as categories.

7. Collapsing categories into broader buckets

A column with dozens of rare values is a problem for both charts and models. The fix is a mapping dictionary passed to .replace, collapsing fine-grained values into a handful of buckets before conversion.

import pandas as pd
pets = pd.DataFrame({"coat": ["short", "medium", "long", "curly", "wire",
                              "short", "long", "hairless", "medium", "short"]})
collapse = {"medium": "medium-long", "long": "medium-long",
            "curly": "other", "wire": "other", "hairless": "other"}
pets["coat_grouped"] = pets["coat"].replace(collapse).astype("category")
print(pets["coat_grouped"].value_counts())
# short          4
# medium-long    4
# other          3  ... wait: 3? count again below
print(pets["coat_grouped"].value_counts(normalize=True).round(2))

Six original coat types become three usable groups, and value_counts(normalize=True) shows each bucket’s share rather than its raw count. Collapsing before modelling keeps one-hot encoding (Example 9) from exploding into a column per rare value, and keeps every bucket large enough to actually learn from.

8. Grouping and counting with categoricals

Grouping by an ordered categorical returns groups in category order, not alphabetical order, which makes summaries read correctly on their own. And value_counts has two arguments worth knowing: normalize for shares and dropna=False to keep missing values visible.

import pandas as pd
import numpy as np
orders = pd.DataFrame({
    "size": pd.Categorical(
        ["large", "small", np.nan, "medium", "small", "large", "small"],
        categories=["small", "medium", "large"], ordered=True),
    "amount": [90, 20, 35, 45, 25, 80, 15],
})
print(orders.groupby("size", observed=False)["amount"].mean())
# small     20.0
# medium    45.0
# large     85.0   <- category order, not alphabetical
print(orders["size"].value_counts(normalize=True, dropna=False).round(2))
# small 0.43, large 0.29, medium 0.14, NaN 0.14

The grouped means come out small, medium, large, a readable progression, where an unordered text column would produce large, medium, small alphabetically. And dropna=False surfaces that 14 percent of rows have no size at all, which a default value_counts would quietly hide from you.

9. Encoding for machine learning: codes and dummies

Models need numbers. Ordered categories map naturally to their integer codes, which is label encoding and suits tree models; unordered categories become one column per value with pd.get_dummies, which is one-hot encoding and suits linear models.

import pandas as pd
cars = pd.DataFrame({
    "condition": pd.Categorical(["fair", "good", "excellent", "good"],
                                categories=["fair", "good", "excellent"],
                                ordered=True),
    "transmission": pd.Categorical(["manual", "automatic", "manual", "automatic"]),
    "price": [4500, 7800, 12100, 8200],
})
# label encoding: ordered category -> its codes (keep the mapping!)
cars["condition_code"] = cars["condition"].cat.codes
mapping = dict(enumerate(cars["condition"].cat.categories))
print(mapping)   # {0: 'fair', 1: 'good', 2: 'excellent'}
# one-hot encoding: unordered category -> one boolean column per value
encoded = pd.get_dummies(cars, columns=["transmission"])
print(encoded[["price", "condition_code",
               "transmission_automatic", "transmission_manual"]])

The condition codes 0, 1, 2 genuinely mean something because the category is ordered, fair really is less than excellent, so a tree model can split on them sensibly. Transmission has no order, so giving it codes would invent a fake ranking; get_dummies sidesteps that with a column per value. And always save the mapping dictionary, because it is the only way to translate predictions back into words.

10. Putting it together: visualising categoricals with catplot

The finale is seaborn’s catplot, the categorical plotting workhorse. One function covers boxes, bars, and counts via its kindargument, with hue adding a second categorical dimension.

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
n = 200
reviews = pd.DataFrame({
    "traveler_type": pd.Categorical(rng.choice(["business", "couple", "family"], n)),
    "stay_length": pd.Categorical(rng.choice(["short", "long"], n)),
    "score": np.clip(rng.normal(7.5, 1.5, n), 1, 10).round(1),
})
sns.catplot(data=reviews, x="traveler_type", y="score",
            hue="stay_length", kind="box")
plt.savefig("reviews_by_type.png", dpi=150, bbox_inches="tight")
print("saved reviews_by_type.png")
# swap kind="box" for "bar", "point", or "count" and rerun

One line produces a boxplot of scores per traveller type, split by stay length within each. The kind argument is the whole API: "count" needs no y at all and just tallies the categories, "bar" shows means with confidence intervals, and "point"connects them. Because the columns are categoricals, the axis order follows the categories, so everything you set up in the earlier examples pays off in the chart.

Work through these and you have the whole article in practice: the code-and-lookup-table machinery, three creation routes, the .cat accessor, ordered comparisons, safe category management, the clean-then-convert pipeline, collapsing rare values, order-aware grouping, both ML encodings, and catplot. The habit that matters most is the sequencing from Examples 6 and 7: clean the strings, collapse the rare values, then convert, because categorical hygiene done up front is what keeps every downstream group, encoding, and chart honest.

See you soon.