Python is not slow. Python loops are slow. This guide is about the difference: how to hand the looping to the built-ins, to itertools, to sets, to NumPy and to pandas, so the work happens in C instead of in the interpreter, and how to measure whether any of it actually helped.
There is one idea underneath almost every technique here. The fastest loop is the one you do not write.
Python has a reputation for being slow next to C or Rust, and it deserves part of it, but the cost is concentrated in a very specific place: the interpreter’s overhead on every single operation. Each i += 1, each names[i], each .append() involves a name lookup, a type check, reference counting and bytecode dispatch. Do that a million times and the overhead dwarfs the actual work.
Built-ins like map, zip and enumerate, along with NumPy arrays and pandas columns, push that loop down into compiled C. The work per element stays the same. The overhead per element nearly vanishes. That is where the speedups of ten times, a hundred times, sometimes a thousand times come from.
So “writing efficient Python” is not a bag of clever tricks. It is a habit of asking, every time you type for, whether something else could do the iterating.
The data used throughout
A board game collection carries the list examples. game_names and game_genres are lists of strings, editions is a list of integers, and complexity and playtest_scores are NumPy arrays, the second one two-dimensional with a row per game. Three friends’ collections, alex_shelf, priya_shelf and sam_shelf, cover the set operations.
The pandas sections use campaigns, a DataFrame of marketing campaigns with columns clicks, impressions, spend, revenue, flagged and month.
1. Measure first, optimise second
Put this section first because everything after it is worthless without it. Nobody’s intuition about performance survives contact with a benchmark. CPU caches, memory layout and interpreter special cases will flip your expectations regularly, and the only reliable move is to write both versions and time them.
In Jupyter or IPython, %timeit does the whole job:
%timeit [num for num in range(51)]%timeit [*range(51)]
It runs the expression many times, picks a sensible repeat count on its own, handles the warm-up, and reports something like 2.45 us +- 12 ns per loop, mean +- std. dev. of 7 runs. The first number is the average time for one execution. The second is the standard deviation across the repeat runs, and it is the number people ignore. If the deviation is a large fraction of the mean, your benchmark is noisy: close some applications, restart the kernel and run it again before believing anything.
A few variants are worth knowing. %%timeit with two percent signs at the top of a cell times the whole cell instead of one line. -n sets how many times the code runs per loop and -r sets how many loops to repeat, which is useful when the default guess is wrong for very fast or very slow code. -o returns the result as an object so you can compare timings programmatically instead of reading them off the screen.
Outside a notebook, %timeit does not exist. Use the standard library instead, either python -m timeit "your code" from the shell or time.perf_counter() around the section you care about.
One thing to keep in perspective. Most code does not need any of this. Optimising a function that runs once on 200 rows is a waste of an afternoon. Measure, find the part that actually costs you, and optimise that.
2. Pythonic code is not just prettier
C-style Python works and it is slow. Compare:
# Non-Pythonici = 0long_names = []while i < len(game_names): if len(game_names[i]) >= 6: long_names.append(game_names[i]) i += 1# Pythoniclong_names = [name for name in game_names if len(name) >= 6]
The comprehension does the same work in one statement. It runs its loop in C, it skips the repeated attribute lookup for .append, and it preallocates the result list rather than growing it by hand. In practice that lands somewhere around a third faster for a loop this simple, and the gap widens as the loop body gets more trivial, because the overhead is a bigger share of the total.
The speed is the secondary benefit. The comprehension has no counter to get wrong, no len() re-evaluated on every pass, and no chance of an off-by-one. Shorter code with fewer moving parts is code with fewer bugs.
There is a memory variant worth knowing too. Swap the square brackets for round ones and you get a generator expression, which produces values one at a time instead of building the whole list:
total = sum(len(name) for name in game_names)
Nothing is materialised. If you only need to consume the values once, this is strictly better than building a list to throw away, and on large data the memory difference is the whole ball game.
3. range: numbers without a list
numbers = range(6)print(type(numbers)) # <class 'range'>as_list = list(numbers) # [0, 1, 2, 3, 4, 5]odds = [*range(1, 12, 2)] # [1, 3, 5, 7, 9, 11]
range is lazy. It does not build a sequence, it stores three integers, the start, the stop and the step, and produces values on demand as you iterate. That is why range(10**9) is instant and harmless: sys.getsizeof(range(10**9)) returns 48 bytes, the same as range(6), while the equivalent list of a billion integers would need tens of gigabytes.
The general form is range(start, stop, step), where stop is exclusive and step defaults to 1. range(5) gives 0 through 4, range(2, 8)gives 2 through 7, range(0, 10, 2) gives the even numbers below 10, and a negative step counts down, so range(10, 0, -1) gives 10 down to 1.
When you genuinely need a list, both list(range(...)) and [*range(...)] work. The unpacking form is marginally faster because it skips the global name lookup for list, but the difference is tens of nanoseconds and it varies between Python versions. Do not restructure code for it. Do measure it yourself if you have a loop hot enough to care, which is the point of section 1.
4. enumerate: index and value together
Any time you catch yourself writing for i in range(len(things)):, this is the replacement.
for i, name in enumerate(game_names): print(i, name)numbered = [*enumerate(game_names, 1)]
enumerate yields (index, value) pairs, which you unpack straight in the loop header. The version with range(len(...)) is not just longer, it also pays for an extra game_names[i] index lookup on every pass, and it gives you a counter you can get wrong.
The optional second argument sets the starting number, so enumerate(game_names, 1) counts from 1 instead of 0. That is what you want whenever the numbers are going to be read by a person rather than used as offsets.
Like range, enumerate is lazy. It returns an iterator, so [*enumerate(...)] or a for loop is what actually makes it produce anything.
5. map: a function over everything
upper_names = [*map(str.upper, game_names)]
map(func, iterable) applies a function to every element, lazily. Calling it does no work at all; the work happens when you iterate or materialise.
The str.upper reference is a nice idiom. Rather than writing lambda s: s.upper(), you pass the function that lives on the str class itself, and str.upper(s) does exactly what s.upper() does. It saves a layer of Python-level call on every element.
When to use map and when to use a comprehension is mostly a style question with a small performance tail. map has the edge when you already have a named function to apply. A comprehension wins when you need an expression, a filter, or access to the index, because map cannot do any of those without help. The one form to avoid is map with an inline lambda: map(lambda x: x * 2, values) is both slower and harder to read than [x * 2 for x in values], because you have reintroduced the Python-level call you were trying to skip. If you are writing a lambda, write a comprehension instead.
6. zip: iterating in parallel
paired = [*zip(game_names, editions)]
zip walks several iterables at once and yields tuples of their corresponding elements. It is lazy like the rest.
Two things about it catch people out. The first is truncation: zip stops as soon as the shortest input runs out. Zip a 200-item list against a 50-item list and you get 50 pairs and no warning, which is a silent, data-losing bug. Python 3.10 added strict=True, which raises a ValueError when the lengths differ, and it is worth turning on by default:
paired = [*zip(game_names, editions, strict=True)]
If you actually want the longer behaviour, with a filler for the missing values, itertools.zip_longest does that.
The second is that zip also unzips. Because * unpacks a list of tuples into separate arguments, feeding paired data back through zip splits it apart again:
pairs = [(1, "a"), (2, "b"), (3, "c")]numbers, letters = zip(*pairs) # (1, 2, 3) and ('a', 'b', 'c')
7. Literals beat constructors
[] and list() produce the same empty list. The literal is a single bytecode instruction that the parser emits directly. The call has to look up the name list, which could in principle have been rebound, fetch the type object and invoke it.
The gap is tens of nanoseconds, so this only matters inside a hot loop that creates millions of containers. The rule is still worth having, because the literal is also shorter: prefer [], {} and () over list(), dict() and tuple().
Two exceptions. There is no literal for an empty set, because {} is already an empty dict, so set() is the only option. And the constructor form is a different operation once you give it an argument: list("hello") returns ['h', 'e', 'l', 'l', 'o'], which no literal will do for you.
8. Sets: the biggest algorithmic win here
Everything above shaves a constant factor. Sets change the complexity class, which is a different order of improvement entirely.
Membership testing
A list answers x in lst by scanning it from the start until it finds a match or runs out. That is O(n). A set hashes x, jumps straight to the matching bucket and checks one or two entries. That is O(1), regardless of how many elements the set holds.
sam_set = set(sam_shelf)"Carcassonne" in sam_shelf # linear scan"Carcassonne" in sam_set # hash lookup
Building the set costs one O(n) pass. After that every check is free. So the moment you are testing membership against the same collection more than once or twice, convert it first.
The version of this that really hurts is nested:
# O(n * m)shared = [g for g in alex_shelf if g in priya_shelf]# O(n + m)priya_set = set(priya_shelf)shared = [g for g in alex_shelf if g in priya_set]
With 100,000 items on each side, the first form is ten billion comparisons and the second is a hundred thousand hash lookups. That is the difference between a coffee break and no perceptible delay.
Tuples behave like lists here, since they scan too. Dictionary key lookup is O(1) like a set, because a dict is a hash table with values attached.
Deduplication
The naive version is a textbook accident:
def find_unique(data): uniques = [] for item in data: if item not in uniques: # O(n) scan of a growing list uniques.append(item) return uniques
That if item not in uniques scans a list that grows on every iteration, so the whole function is O(n squared). For 10,000 items it performs on the order of 100 million comparisons. set(data) does the same job in 10,000 operations.
The cost of the set is that it does not keep the original order. When order matters, dict.fromkeys gives you both properties, because dicts have preserved insertion order since Python 3.7:
unique_ordered = list(dict.fromkeys(game_names))
The general lesson generalises past dedup. Any time you write if x not in some_list inside a loop, you have built an O(n squared) algorithm by accident.
Set algebra
Once the data is in sets, comparisons come free:
alex_set = set(alex_shelf)priya_set = set(priya_shelf)both = alex_set & priya_set # intersection, in botheither = alex_set | priya_set # union, in at least onealex_only = alex_set - priya_set # difference, in the first onlyexactly_one = alex_set ^ priya_set # symmetric difference
Each operator has a method form as well, .intersection(), .union(), .difference() and .symmetric_difference(), which read better in isolation. The operators read better when you are chaining several together. All of them are hash-based and linear in the total size, where the list equivalents are quadratic.
9. Counter and itertools
Counting is common enough to have a dedicated C-optimised tool:
from collections import Countergenre_counts = Counter(game_genres)genre_counts.most_common(3)genre_counts["Strategy"] # returns 0 for a missing key, no KeyError
Counter takes any iterable and returns a dict subclass mapping each value to its count. It beats the manual counts[x] = counts.get(x, 0) + 1 pattern on both speed and length. Because it accepts any iterable, you can feed it a generator expression and skip the intermediate list entirely: Counter(name[0] for name in game_names) counts starting letters without building a list of them.
itertools covers the combinatorial patterns:
from itertools import combinationspairs = [*combinations(game_genres, 2)]quads = [*combinations(game_genres, 4)]
combinations(iterable, r) yields every r-element subset, without repetition and without regard to order, so combinations(["A", "B", "C"], 2) gives ("A","B"), ("A","C") and ("B","C") and never ("B","A"). That is the right tool for round-robin pairings, for comparing every item against every other, and for enumerating feature subsets.
The rest of the family covers the neighbouring cases. permutations(iterable, r) is the version where order matters. combinations_with_replacement allows an element to repeat within a selection. product(*iterables) gives the Cartesian product, which is what a nested loop over several lists produces. chain(*iterables) flattens several iterables end to end. cycle repeats one forever, and islice takes a slice of any iterator without materialising it first. All of them are lazy and implemented in C.
10. NumPy: where the real speed lives
A Python list of a million numbers is a million pointers to a million separate objects, each carrying its own type information, scattered across memory. Adding one to every element means dereferencing each pointer, checking a type, doing the arithmetic and allocating a fresh object for the result. Hundreds of nanoseconds per element.
A NumPy array of a million numbers is one contiguous block of raw bytes, all the same type. Adding one to every element hands the whole block to a C routine that walks it in a tight loop, often using the CPU’s vector instructions to process several values per cycle. Single-digit nanoseconds per element.
That memory layout is the entire explanation for the speed, and it is why the rule of thumb is so blunt: if the data is homogeneous and numeric and you are about to write a for loop, use an array instead.
playtest_scores[1, :] # row 1, every columnplaytest_scores[:, 2] # every row, column 2playtest_scores[playtest_scores > 6] # every element above 6doubled = playtest_scores * 2 # elementwise, in Cplaytest_scores[:, 2] += 1 # whole column at once
Two of those deserve names. Broadcasting is what makes playtest_scores * 2 work: the scalar is applied across the whole array without any loop on your side. Boolean masking is what makes the filter work: playtest_scores > 6 is itself an array of True and False the same shape as the original, and indexing with it returns only the matching elements.
Boolean masking is the vectorised form of the most common loop in all of Python:
# The loopstrategy_games = []for name, genre in zip(game_names, game_genres): if genre == "Strategy": strategy_games.append(name)# The masknames_np = np.array(game_names)genres_np = np.array(game_genres)strategy_games = names_np[genres_np == "Strategy"]
For a hundred elements the difference is invisible. For ten thousand it is roughly a hundredfold. For a million the loop takes seconds and the mask takes milliseconds. Whenever you see for x in data: if condition: result.append(x), there is a mask waiting to replace it.
The axis argument
This is the part of NumPy that confuses everyone at first, and there is a mental shortcut that fixes it. axis=N names the axis that disappears.
playtest_scores.sum(axis=1) # collapses the columns, one total per rowplaytest_scores.mean(axis=0) # collapses the rows, one average per column
So axis=0 collapses down through the rows and leaves you one value per column, and axis=1 collapses across the columns and leaves you one value per row. Ask which dimension you want to get rid of, and that is the number you pass.
One caution before you reach for NumPy everywhere. It only wins when the data is already an array. np.sum(some_python_list) has to convert the list first, and it ends up slower than the plain built-in sum(). NumPy is a payoff for keeping your numbers in arrays, not a spell you cast on lists.
11. Loop hygiene
Two habits that cost nothing and pay constantly.
Hoist anything that does not change
# Beforefor genre, count in genre_counts.items(): total = len(game_genres) # recomputed every pass share = round(count / total * 100, 2)# Aftertotal = len(game_genres)for genre, count in genre_counts.items(): share = round(count / total * 100, 2)
A C or Java compiler would lift that len() out of the loop for you. CPython will not: it dutifully recomputes it on every iteration, because in principle the list could have changed. So this one is on you, and it is one of the easiest wins available.
What to look for when you read a loop body: len() on a collection that is not being modified, dictionary lookups on a constant key, aggregations like .mean() or .sum() on data that is not changing, and any function call whose arguments do not involve the loop variable. Ask of each line whether it computes the same answer every time round. If it does, lift it out.
Batch your conversions
# Converting inside the looprows = []for i, pair in enumerate(pairs, 1): rows.append(list((i,) + pair))# Collecting, then converting oncetuples = []for i, pair in enumerate(pairs, 1): tuples.append((i,) + pair)rows = [*map(list, tuples)]
For a cheap conversion like list() on a short tuple this barely registers. It matters when the conversion is expensive, which is exactly the case when a NumPy array, a pandas object or an external library is involved, because there the fixed per-call overhead swamps the actual work. Doing it once on the whole batch pays that overhead once.
The broader habit is to separate computation from conversion. Do the iteration in whatever form is natural, then convert in bulk at the end.
A related trap: building strings in a loop
# O(n squared): every += builds a whole new stringreport = ""for name in game_names: report += name + "\n"# O(n)report = "\n".join(game_names)
Strings are immutable, so += allocates a new string and copies the old contents every single time. str.join measures the total length once, allocates once and copies once.
12. A worked example: vectorised z-scores
Standardising a column is a good test case, because the naive version manages to make both classic mistakes at once.
# Beforescored = []for name, value in zip(game_names, complexity): mean = complexity.mean() # invariant, recomputed n times sd = complexity.std() # invariant, recomputed n times z = (value - mean) / sd # one Python float op per element scored.append((name, value, z))
Two problems. The mean and standard deviation do not depend on the loop variable, so for a thousand games that is two thousand unnecessary aggregations, each of which walks the whole array. And the arithmetic runs one element at a time at Python speed.
# Aftermean = complexity.mean()sd = complexity.std()z_scores = (complexity - mean) / sdscored = [*zip(game_names, complexity, z_scores)]outliers = [(name, value, z) for name, value, z in scored if z > 2]
The formula is unchanged:
/math z = \frac{x – \bar{x}}{s}
The aggregations happen once, and the subtraction and division run across the whole array in C through broadcasting. On any meaningful amount of data this is easily a hundredfold difference.
Two details worth noting. The final filter stays a plain comprehension, and that is fine: it is a cheap filter over tuples, not numeric work, and the hybrid approach of vectorising the heavy part while keeping Python for the light part is the normal shape of good code. And ndarray.std() defaults to ddof=0, the population form, while pandas .std() defaults to ddof=1. If you want the sample standard deviation, pass ddof=1 explicitly.
13. pandas: four ways to walk a DataFrame, in order of badness
.iterrows()
margins = []for i, row in campaigns.iterrows(): margins.append(row["revenue"] - row["spend"])campaigns["margin"] = margins
.iterrows() yields (index, row) pairs where the row is a Series, so you access columns by name like a dictionary. It is the friendliest of the options and much the slowest, because it constructs a fresh Series object for every row, with all the dtype handling and label bookkeeping that implies.
There is a subtler problem. A Series holds one dtype, so a row spanning integer and float columns gets coerced to a common type on the way through. Your integers can quietly arrive as floats.
Use it when the frame is small, when you are prototyping and do not care, or when you genuinely need the row as a Series. Otherwise do not.
.itertuples()
margins = []for row in campaigns.itertuples(): margins.append(row.revenue - row.spend)campaigns["margin"] = margins
.itertuples() yields namedtuples instead, so columns come through attribute access as row.revenue and the index as row.Index. It skips the Series construction entirely, which makes it several times faster, and it preserves each column’s dtype.
The one quirk is that column names have to be valid Python identifiers to work as attributes. A column called total spendgets mangled into a positional name like _3. Passing name=None yields plain tuples, which is faster still if you are happy indexing by position, and index=False drops the index from each tuple.
If you are going to loop a DataFrame at all, loop it with this.
.apply(axis=1)
campaigns["margin"] = campaigns.apply( lambda row: row["revenue"] - row["spend"], axis=1)
This is the one that misleads people. The syntax looks like a bulk operation, so it reads as though it were vectorised. It is not. With axis=1, .apply walks the frame row by row and calls your Python function on each one, which puts it in the same performance class as .itertuples(), not in the same class as a column operation.
It earns its place when you have a genuinely row-level custom function that cannot be expressed as column arithmetic. It does not earn its place for anything that is plain arithmetic, comparison or boolean logic on columns.
Column operations and .to_numpy()
campaigns["margin"] = campaigns["revenue"] - campaigns["spend"]campaigns["ctr"] = campaigns["clicks"] / campaigns["impressions"]
This is the destination. A pandas column is a NumPy array with labels attached, so arithmetic between columns runs elementwise in C on the whole thing at once. The seven-line loop becomes one line that is both shorter and a hundred times faster.
For a custom function, extract the raw arrays and call it on those:
def expected_share(revenue, spend): return np.round(revenue ** 2 / (revenue ** 2 + spend ** 2), 2)campaigns["share"] = expected_share( campaigns["revenue"].to_numpy(), campaigns["spend"].to_numpy(),)
The key move is that expected_share was written using nothing but arithmetic operators, so it works unchanged whether you hand it two scalars or two arrays. NumPy’s operators handle both. Designing functions this way, with plain arithmetic and no Python-only constructs, means they vectorise for free when you need them to.
You will see .values in older code and it still works, but .to_numpy() is the form pandas now recommends and it is clearer about what you are getting.
The awkward case is conditional logic, because if and else do not survive the trip to arrays. The replacements are np.where(condition, value_if_true, value_if_false) for a single branch, boolean masking for filtering, and np.select when there are several branches.
Roughly what the four cost on the same job:
| Approach | Relative speed | Why |
|---|---|---|
.iterrows() | Baseline | Builds a Series per row, coerces dtypes |
.itertuples() | Around 3 to 5 times faster | Namedtuples, no Series construction |
.apply(axis=1) | Comparable to .itertuples() | Still row by row, plus dispatch overhead |
Column ops or .to_numpy() | 100 to 1000 times faster | No Python-level loop at all |
The gaps between the first three are a constant factor. The gap to the fourth is a different kind of thing, and it is the one worth chasing.
One more pandas habit while we are here: when you find yourself looping over groups of rows, .groupby() is the vectorised answer. campaigns.groupby("month")["revenue"].sum() does in one call what a loop over unique months with a filter inside would do slowly.
14. The decision tree
WORKING WITH LISTS need index and value -> enumerate() iterating two lists together -> zip(), with strict=True applying a function -> map() or a comprehension counting occurrences -> Counter() testing membership repeatedly -> convert to a set first deduplicating -> set(), or dict.fromkeys() to keep order pairings and orderings -> itertools homogeneous numbers -> NumPy array consuming values once -> generator expression, not a listWORKING WITH DATAFRAMES arithmetic between columns -> df["a"] - df["b"], directly custom function per row -> write it with arithmetic only, then call it on .to_numpy() conditional logic -> np.where, masking, or np.select aggregating by category -> .groupby() must loop -> .itertuples(), never .iterrows()INSIDE ANY LOOP value does not change -> hoist it out same conversion every pass -> collect first, convert once at the end "if x not in some_list" -> make it a set, you have an O(n squared) bug building a string with += -> collect and use "".join()BEFORE ANY OF IT %timeit both versions -> do not trust your intuition, or mine
Three sentences hold the whole thing together. The interpreter is what costs you, so the goal of every technique here is to do fewer operations at Python level and more at C level. Constant-factor wins come from the built-ins and from NumPy, while the order-of-magnitude wins come from replacing an accidental O(n squared) with a set or a mask. And none of it counts until %timeit says so, because performance intuition is wrong often enough that guessing is not a strategy.
Write the clear version first. Measure it. Then, and only where the measurement points, take the loop out.
Thanks, Andrei.
[…] Writing Efficient Python Code […]