Exploratory Data Analysis in Python: 10 Code-Along Examples

Learn EDA by cleaning a broken dataset. Ten copy-and-run examples covering first inspection, range checks, missing values, group imputation, type repair, binning, crosstabs, transform, outliers, and distribution checks.

The exploratory data analysis article covers the step that separates analysts who get it right from those who do not: understanding a dataset before trusting it. This workbook is that process end to end on one deliberately broken table, and the examples are cumulative rather than independent, so the frame you finish Example 1 with is the frame Example 2 starts from, exactly as a real session works. A stray age of 214 and a fare of 9,450 are planted in the data from the beginning; you will see them distort the summary statistics in Example 1, poison a group standard deviation in Example 8, and finally get caught in Example 9, which is the whole argument for doing this work before modelling anything. The idea that ties it together arrives in the last example. The dataset is generated in code with a fixed seed, so your numbers will match the ones printed here.

Paste this once to create the working dataset:

import numpy as np
import pandas as pd
rng = np.random.default_rng(7)
n = 200
carriers = rng.choice(["Northwind", "Bluejay", "Kestrel"], size=n, p=[0.5, 0.3, 0.2])
base = {"Northwind": 180, "Bluejay": 95, "Kestrel": 320}
fares = np.array([rng.normal(base[c], base[c] * 0.18) for c in carriers]).round(2)
bookings = pd.DataFrame({
    "booking_id": range(1000, 1000 + n),
    "carrier": carriers,
    "destination": rng.choice(["Lisbon", "Oslo", "Athens", "Dublin"], size=n),
    "cabin": rng.choice(["economy", "premium", "business"], size=n, p=[0.7, 0.2, 0.1]),
    "passenger_age": rng.integers(18, 78, size=n),
    "fare": fares,
    "flight_hrs": [f"{h}h" for h in rng.integers(1, 15, size=n)],
    "remarks": [None] * n,
})
# the problems a real extract would arrive with
bookings.loc[rng.choice(n, 14, replace=False), "fare"] = np.nan
bookings.loc[rng.choice(n, 3, replace=False), "cabin"] = np.nan
bookings.loc[5, "passenger_age"] = 214
bookings.loc[12, "fare"] = 9450.00

1. The first look

Four calls, under a minute, and you have a working mental model of a dataset you have never seen.

print(bookings.head(3).to_string())
bookings.info()
print(bookings[["passenger_age", "fare"]].describe().round(2))
print(bookings["cabin"].value_counts(dropna=False))

   booking_id  carrier destination    cabin  passenger_age    fare flight_hrs remarks
0        1000  Bluejay        Oslo  premium             76  110.76        10h    None
1        1001  Kestrel      Lisbon  economy             51  293.76        11h    None
2        1002  Bluejay      Lisbon  economy             44     NaN        10h    None

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 200 entries, 0 to 199
Data columns (total 8 columns):
 #   Column         Non-Null Count  Dtype
---  ------         --------------  -----
 0   booking_id     200 non-null    int64
 1   carrier        200 non-null    object
 2   destination    200 non-null    object
 3   cabin          197 non-null    object
 4   passenger_age  200 non-null    int64
 5   fare           186 non-null    float64
 6   flight_hrs     200 non-null    object
 7   remarks        0 non-null      object
dtypes: float64(1), int64(2), object(5)

       passenger_age     fare
count         200.00   186.00
mean           49.36   234.17
std            20.42   685.02
min            18.00    63.93
25%            33.75   108.33
50%            47.00   176.41
75%            64.00   229.39
max           214.00  9450.00

cabin
economy     140
premium      43
business     14
NaN           3
Name: count, dtype: int64

Four problems are already visible and none of them needed a chart. remarks has zero non-null values, so it is a column of nothing. cabin and fare have fewer non-nulls than the 200 rows, which is the missing-data report. flight_hrs is typed object, meaning pandas is storing “10h” as a string and no arithmetic will work on it. And the describe block is shouting: a maximum age of 214, a fare maximum of 9,450 against a 75th percentile of 229, and a standard deviation of 685 on a mean of 234, which is the classic signature of a single extreme value. Note dropna=False on the last call, without which value_counts hides the missing category entirely and the proportions quietly describe only the rows that happen to be complete.

2. Range checks and the tilde filter

The cheapest validation is asking whether the smallest and largest values in a column are physically possible.

print("age  min/max:", bookings["passenger_age"].min(), "/", bookings["passenger_age"].max())
print("fare min/max:", bookings["fare"].min(), "/", bookings["fare"].max())
impossible = bookings[bookings["passenger_age"] > 120]
print(impossible[["booking_id", "passenger_age"]].to_string(index=False))
not_kestrel = ~bookings["carrier"].isin(["Kestrel"])
print("rows kept by ~isin:", not_kestrel.sum(), "of", len(bookings))
print(bookings[not_kestrel]["carrier"].unique())

age  min/max: 18 / 214
fare min/max: 63.93 / 9450.0
rows failing the range check:
 booking_id  passenger_age
       1005            214
rows kept by ~isin: 156 of 200
['Bluejay' 'Northwind']

One booking claims a 214-year-old passenger, and finding it took two method calls. That is the point of a range check: it is not sophisticated, it costs nothing, and it catches the class of error that will otherwise sit quietly in your data until it appears in a chart in front of a client. Decide what to do with it deliberately, since a value that is impossible rather than merely extreme is usually a data-entry fault, which means correcting it, dropping the row, or setting it to NaN and treating it as missing, but never leaving it. The ~ operator underneath flips a boolean Series, so ~isin([...]) reads as “keep every row whose carrier is not in this list” and is the standard way to express exclusion.

3. How much is missing, and where

Counting missing values is the easy part. Turning the count into a proportion is what makes the decision for you.

missing = bookings.isna().sum()
print(missing[missing > 0])
share = (bookings.isna().sum() / len(bookings)).round(4)
print(share[share > 0])
print("columns that are entirely empty:", bookings.columns[bookings.isna().all()].tolist())

cabin        3
fare        14
remarks    200
dtype: int64
cabin      0.015
fare       0.070
remarks    1.000
dtype: float64
columns that are entirely empty: ['remarks']

Three columns with gaps and three different answers. remarks is 100 percent missing and has no information in it at all, so it goes. cabin is missing in 1.5 percent of rows, comfortably under the rough five percent threshold where dropping the affected rows is unlikely to bias anything. fare is missing in seven percent, which is enough that throwing those rows away starts to cost you, and enough that you should ask whether the missingness is random before filling it. That question matters more than the technique: if fares are missing mostly for one carrier, then any fill you choose is inventing data about that carrier, and the honest move is to say so in the write-up rather than to pick a cleverer imputation.

4. Dropping and imputing, group by group

Three different problems from Example 3 get three different treatments, and the imputation is the one worth being careful about.

before = len(bookings)
bookings = bookings.drop(columns=["remarks"])
threshold = len(bookings) * 0.05
light = bookings.columns[bookings.isna().sum() <= threshold]
bookings = bookings.dropna(subset=light)
print(f"rows {before} -> {len(bookings)} after dropping light-missing rows")
print(bookings.isna().sum()[bookings.isna().sum() > 0])
group_median = bookings.groupby("carrier")["fare"].median()
print("global median:", round(bookings["fare"].median(), 2))
print(group_median.round(2))
bookings["fare"] = bookings["fare"].fillna(bookings["carrier"].map(group_median))
print("missing fares after group imputation:", int(bookings["fare"].isna().sum()))

rows 200 -> 197 after dropping light-missing rows
still missing:
fare    14
dtype: int64
global median: 178.11
group medians:
carrier
Bluejay       93.95
Kestrel      315.16
Northwind    181.25
Name: fare, dtype: float64
missing fares after group imputation: 0

Compare the global median of 178 with the group medians of 94, 315 and 181. Filling every missing fare with 178 would have inflated every Bluejay booking by roughly 85 pounds and cut every Kestrel booking by 137, inventing a pattern where the carriers look more alike than they are, which is exactly the structure you were about to analyse. The group version fills each row from its own carrier’s median and preserves the differences. The mechanics are worth reading closely: groupby().median() produces a Series indexed by carrier, .map() looks each row’s carrier up in it, and fillna uses the result only where the fare is missing. The dropping step above uses a computed column list so the five percent rule is applied by the code rather than by hand.

5. Repairing types

A numeric column stored as text is the most common type problem, and the fix is always clean then cast.

print(bookings.dtypes)
print("flight_hrs sample:", bookings["flight_hrs"].head(3).tolist())
bookings["flight_hrs"] = bookings["flight_hrs"].str.replace("h", "", regex=False)
bookings["flight_hrs"] = pd.to_numeric(bookings["flight_hrs"], errors="coerce")
print("after clean-and-cast:", bookings["flight_hrs"].dtype,
      "| mean", round(bookings["flight_hrs"].mean(), 2))
dirty = pd.Series(["12", "8", "n/a", "5"])
print("errors='coerce' turns junk into NaN:", pd.to_numeric(dirty, errors="coerce").tolist())

booking_id         int64
carrier           object
destination       object
cabin             object
passenger_age      int64
fare             float64
flight_hrs        object
dtype: object
flight_hrs sample: ['10h', '11h', '10h']
after clean-and-cast: int64 | mean 7.42
errors='coerce' turns junk into NaN: [12.0, 8.0, nan, 5.0]

One suffix was enough to make an entire column unusable, since "10h" cannot be averaged, compared or plotted, and pandas will not guess what you meant. Stripping the unit with .str.replace and casting with pd.to_numeric fixes it in two lines. The errors="coerce" argument is the part to remember: without it, a single unparseable value anywhere in a 200,000-row column raises and the whole operation fails, while with it the bad values become NaN, which you can then count and deal with as missing data using Example 3’s tools. That turns an all-or-nothing failure into a measurable one, which is almost always what you want during exploration.

6. Cardinality, and binning with np.select

Object columns need a different first question from numeric ones: not what is the range, but how many distinct values are there.

non_numeric = bookings.select_dtypes("object")
for col in non_numeric.columns:
    print(f"{col:12s} {non_numeric[col].nunique():>4} unique")
conditions = [
    bookings["flight_hrs"] <= 4,
    bookings["flight_hrs"] <= 9,
    bookings["flight_hrs"] <= 15,
]
labels = ["Short-haul", "Medium-haul", "Long-haul"]
bookings["haul_type"] = np.select(conditions, labels, default="Ultra-long")
print(bookings["haul_type"].value_counts())

carrier         3 unique
destination     4 unique
cabin           3 unique
haul_type
Long-haul      70
Medium-haul    69
Short-haul     58
Name: count, dtype: int64

Three columns, all low cardinality, all genuinely categorical and all ready to be encoded for a model later. High cardinality would have told you something different, since a text column with a thousand distinct values in a thousand rows is an identifier or free text rather than a category, and it needs extraction or dropping rather than encoding. The binning underneath is np.select, which is a vectorised if-elif-else evaluated over the whole column at once and far faster than looping or applying a function row by row. The order of the conditions matters exactly as it would in an if-chain, because the first true condition wins, which is why the overlapping thresholds read correctly rather than needing ranges.

7. Two-way views with crosstab

A crosstab answers questions about two categorical columns at once, and adding a third column turns each cell into an aggregation.

print(pd.crosstab(bookings["carrier"], bookings["cabin"]))
print(pd.crosstab(bookings["carrier"], bookings["cabin"],
                  values=bookings["fare"], aggfunc="mean").round(0))
print(pd.crosstab(bookings["carrier"], bookings["cabin"], normalize="index").round(3))

cabin      business  economy  premium
carrier
Bluejay           1       46       12
Kestrel           4       30       10
Northwind         9       64       21
cabin      business  economy  premium
carrier
Bluejay        94.0     95.0     90.0
Kestrel       385.0    303.0    323.0
Northwind     178.0    324.0    176.0
cabin      business  economy  premium
carrier
Bluejay       0.017    0.780    0.203
Kestrel       0.091    0.682    0.227
Northwind     0.096    0.681    0.223

Three views of the same pair of columns. The counts show the shape of the data and the thin cells, with one Bluejay business booking, meaning any average in that cell rests on a single row and should not be quoted. The mean fares show carrier differences dominating cabin differences, and one number that does not fit the pattern: Northwind economy at 324 is higher than Northwind business at 178, which is not how airlines work and is your outlier making itself felt again. The normalised version divides each row by its own total, which is the right comparison when the groups are different sizes, and it shows that the three carriers actually sell a very similar cabin mix once you stop comparing raw counts.

8. agg collapses, transform broadcasts

Both compute a group statistic. They differ in what shape comes back, and that difference is the basis of most feature engineering.

summary = bookings.groupby("carrier").agg(
    bookings=("booking_id", "count"),
    mean_fare=("fare", "mean"),
    std_fare=("fare", "std"),
).round(2)
print(summary)
bookings["carrier_mean_fare"] = bookings.groupby("carrier")["fare"].transform("mean").round(2)
bookings["vs_carrier"] = (bookings["fare"] - bookings["carrier_mean_fare"]).round(2)
print("shape after transform:", bookings.shape)
print(bookings[["carrier", "fare", "carrier_mean_fare", "vs_carrier"]].head(5).to_string(index=False))

           bookings  mean_fare  std_fare
carrier
Bluejay          59      93.88     14.93
Kestrel          44     314.56     53.14
Northwind        94     277.18    956.80

shape after transform: (197, 10)
  carrier   fare  carrier_mean_fare  vs_carrier
  Bluejay 110.76              93.88       16.88
  Kestrel 293.76             314.56      -20.80
  Bluejay  93.95              93.88        0.07
Northwind 139.61             277.18     -137.57
Northwind 207.92             277.18      -69.26

Look at Northwind’s standard deviation: 956.80, against 14.93 and 53.14 for the other two carriers, on a mean fare that is supposed to be the middle of the three. One row is doing that, and this is the moment in a real session where you stop and investigate rather than carrying on. The two tools around it are worth separating clearly. agg collapses to one row per group and is what you print or report, and the named form used here is worth preferring because it labels its own output columns. transform computes the same statistic but broadcasts it back to every row in the group, leaving the frame’s shape unchanged, which is how you build features like “how far is this booking from its carrier’s average”, the vs_carrier column above.

9. Outliers with the IQR rule

The interquartile range gives you a defensible, non-arbitrary definition of an outlier, and it is not affected by the outlier it is looking for.

q1, q3 = bookings["fare"].quantile([0.25, 0.75])
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print(f"Q1 {q1:,.2f}  Q3 {q3:,.2f}  IQR {iqr:,.2f}")
print(f"fences: {lower:,.2f} to {upper:,.2f}")
outliers = bookings[(bookings["fare"] < lower) | (bookings["fare"] > upper)]
print("outliers found:", len(outliers))
print(outliers[["booking_id", "carrier", "fare"]].to_string(index=False))
clean = bookings[(bookings["fare"] >= lower) & (bookings["fare"] <= upper)]
print(f"mean fare with outliers:    {bookings['fare'].mean():,.2f}")
print(f"mean fare without outliers: {clean['fare'].mean():,.2f}")
print(f"rows kept: {len(clean)} of {len(bookings)}")

Q1 107.71  Q3 228.24  IQR 120.53
fences: -73.09 to 409.04
outliers found: 4
 booking_id   carrier    fare
       1007   Kestrel  428.23
       1012 Northwind  9450.00
       1075   Kestrel  419.19
       1145   Kestrel  432.45
mean fare with outliers:    230.63
mean fare without outliers: 179.82
rows kept: 193 of 197

Fifty pounds of difference in the headline average, from four rows out of 197. The rule is Q1 minus 1.5 IQRs and Q3 plus 1.5 IQRs, and it works because quartiles are order statistics that barely move when one value is extreme, unlike a mean and standard deviation which the outlier itself would drag outwards. Read the four hits carefully before removing anything, though, because they are not the same kind of thing: booking 1012 at 9,450 is the planted error, while the three Kestrel fares in the 420s are simply expensive bookings on the expensive carrier and are almost certainly real. That is the judgement the rule cannot make for you, and blanket removal would quietly delete the top of a legitimate distribution, so the honest practice is to flag with the rule and decide by inspection.

10. Distribution checks, and the model

With the repairs done, the same summary statistics from Example 1 tell a completely different story.

print(f"mean   {clean['fare'].mean():>8,.2f}")
print(f"median {clean['fare'].median():>8,.2f}")
print(f"skew   {clean['fare'].skew():>8.2f}")
print(clean.groupby("carrier")["fare"].agg(["count", "mean", "median", "std"]).round(2))
print(pd.cut(clean["fare"], bins=[0, 100, 200, 300, 400]).value_counts().sort_index())

mean     179.82
median   177.10
skew       0.68
           count    mean  median    std
carrier
Bluejay       59   93.88   93.95  14.93
Kestrel       41  306.36  312.07  44.98
Northwind     93  178.55  181.25  31.90
fare
(0, 100]      44
(100, 200]    89
(200, 300]    33
(300, 400]    26
Name: count, dtype: int64

Compare this with Example 1. The mean and median are now 180 and 177, close enough to say the distribution is roughly symmetric, where they were 234 and 176. Skew has come down to 0.68, which is mild. And Northwind’s standard deviation is 31.90 rather than 956.80, so the group that looked wildly volatile was in fact the most ordinary of the three. Nothing about the underlying reality changed between Example 1 and here; only the quality of the data did. The same check belongs in a chart as well, which takes two lines, and the binned counts above are what its bars would show:

import matplotlib.pyplot as plt
clean["fare"].plot(kind="hist", bins=20, edgecolor="white")
plt.title("Fare distribution after cleaning")
plt.show()

The model that ties the ten together is that EDA answers four questions in a fixed order, and doing them out of order is what produces confident wrong answers. First, what is here: the shape, the types and the columns, which is Example 1 and takes a minute. Second, is it possible: range checks, category checks, and anything a domain expert would call absurd, which is Example 2 and is the step most often skipped. Third, what is missing or broken, and what will you do about it: the counting in Example 3, the dropping and group-aware filling in Example 4, and the type repairs in Example 5, where every decision is a decision about what you are willing to invent. And fourth, what does it look like: the distributions, the group comparisons and the outliers in Examples 6 through 10. The order matters because every question depends on the ones above it, and a distribution computed on data that failed question two is not a distribution, it is a description of your errors, which is exactly what Example 1’s summary statistics were.

Work through these and you have the article in practice: the four-call first look and the four problems it exposes; a range check that catches an impossible age; missing values counted as proportions rather than totals; group-aware imputation against a global median that would have flattened the carriers; clean-and-cast type repair with errors="coerce"; cardinality checks and np.select binning; crosstabs in three forms; agg against transform and the standard deviation that gives the outlier away; the IQR rule and the judgement it cannot make; and a before-and-after on the same statistics. The habit that follows is a sequencing one: never compute a statistic you intend to quote until the validation and missing-data steps are done, because the cost of a bad number is not that it is wrong, it is that it is plausible.

Thanks for reading, Andrei.