Comprehensions

Idiomatic iteration — from concise syntax to memory-efficient pipelines

Why

Comprehensions are the most visible marker of Python fluency — any code review in a Python shop will flag a for loop building a list when a comprehension would do. Generators go further: they're the idiomatic way to process large datasets without loading everything into memory, which matters enormously in production data pipelines.

Understanding the memory difference between a list comprehension and a generator expression is a proxy for whether you think about memory at all — a distinction worth internalising in any data-heavy work.

Intuition

A list comprehension builds the entire result upfront and holds it in memory. A generator produces one item at a time, pausing between — it uses O(1) memory regardless of the number of items. The syntax is nearly identical: square brackets vs parentheses.

Think of a generator as a recipe, not a meal. The recipe doesn't cook all the food at once — it gives instructions you follow one step at a time. Only when you ask for the next item does it compute it.

Explanation

A comprehension is a concise way to build a collection from an iterable in one line.

1. Comprehension forms

Comprehensions build collections inline in a single expression. List, dict, and set forms all follow the same [expr for item in iterable if condition] pattern.

# List comprehension — builds full list in memory squares = [x**2 for x in range(10) if x % 2 == 0] # Dict comprehension word_len = {w: len(w) for w in ["cat", "elephant"]} # Set comprehension — deduplicates unique_lengths = {len(w) for w in ["cat", "dog", "elephant"]} # Nested — flatten a 2D list (outer loop first) matrix = [[1, 2], [3, 4], [5, 6]] flat = [x for row in matrix for x in row] # [1, 2, 3, 4, 5, 6]

2. Generator expressions & yield

A generator produces values on demand rather than building the full result upfront — O(1) memory regardless of input size. The yield keyword turns any function into a generator.

# LIST — everything computed immediately, stored in memory lst = [x**2 for x in range(5)] # [0, 1, 4, 9, 16] — all 5 values sitting in memory right now lst[0] # 0 lst[3] # 9 — random access works # GENERATOR — nothing computed yet gen = (x**2 for x in range(5)) # <generator object> — no values computed yet next(gen) # 0 — computes NOW next(gen) # 1 — computes NOW next(gen) # 4 — computes NOW # yield — one value at a time def squares(n): for i in range(n): yield i**2 # Memory comparison import sys lst = [x for x in range(1_000_000)] gen = (x for x in range(1_000_000)) sys.getsizeof(lst) # ~8 MB sys.getsizeof(gen) # 128 bytes

yield turns a function into a generator. Each next() call runs up to the next yield, pauses, and returns the yielded value. Local state is preserved between yields.

yield from delegates to a sub-generator, forwarding all values through — cleaner than a manual for loop when composing generators.

# without yield from — verbose delegation def chain_manual(a, b): for x in a: yield x for x in b: yield x # with yield from — delegates to sub-generator directly def chain(a, b): yield from a yield from b list(chain([1, 2], [3, 4])) # [1, 2, 3, 4] # yield from also works on any iterable (list, range, another generator) def flatten(nested): for item in nested: if isinstance(item, list): yield from flatten(item) # recursive delegation else: yield item list(flatten([1, [2, [3, 4]], 5])) # [1, 2, 3, 4, 5]

3. When to use which

List comprehension

You need to index into the result, iterate multiple times, check length, or pass to something requiring a list. Data fits comfortably in memory.

Generator

Data is large or infinite. Single pass is enough. Feeding a pipeline — sum, max, any, all accept generators natively. Reading files, streaming data from APIs.

← Previous Next →