itertools.groupby groups a sorted sequence

itertools.groupby is one of the most useful tools in the Python standard library for splitting a sequence into runs of related items — without building intermediate lists and without loading everything into memory. But it has two traps that cause bugs even for experienced developers. If your groups ever come out wrong, empty, or “disappear,” this is why.

What groupby actually does

groupby(iterable, key) walks the iterable and yields (group_key, group_iterator) pairs. Items with consecutive equal key(item) values are grouped together. That word — consecutive — is the entire source of the first trap.

from itertools import groupby

logs = [
    ("error", "db down"),
    ("error", "timeout"),
    ("info", "started"),
    ("error", "disk full"),
]

for level, items in groupby(logs, key=lambda r: r[0]):
    print(level, list(items))

Output:

error [('error', 'db down'), ('error', 'timeout')]
info [('info', 'started')]
error [('error', 'disk full')]   # <-- the SAME level, split into two groups!

Trap 1: groupby does NOT sort for you

groupby only groups adjacent equal keys. If your data isn’t already sorted by the key, identical keys that aren’t next to each other will be treated as separate groups. This is the single most common groupby bug.

Wrong (relies on input ordering you don’t control):

# rows read from a DB / API in arbitrary order
rows = [("b", 1), ("a", 2), ("b", 3), ("a", 4)]
for key, items in groupby(rows, key=lambda r: r[0]):
    print(key, list(items))
# b [(b,1)]   a [(a,2)]   b [(b,3)]   a [(a,4)]  <- not grouped!

Right (sort first, by the same key):

rows = [("b", 1), ("a", 2), ("b", 3), ("a", 4)]
rows.sort(key=lambda r: r[0])  # <-- required
for key, items in groupby(rows, key=lambda r: r[0]):
    print(key, list(items))
# a [(a,2), (a,4)]   b [(b,1), (b,3)]

Rule of thumb: if you didn’t sort by the key, assume groupby will surprise you.

Trap 2: group iterators are lazy and one-shot

Each group_iterator is a generator. It is consumed as you iterate it, and it is only valid while the outer loop is still on that group. Two classic failures:

Failure A — iterating a group twice:

for key, items in groupby(rows, key=lambda r: r[0]):
    first = next(items)
    again = next(items)   # ValueError: the generator is already advanced

Failure B — using a group after the outer loop moved on (very common with list comprehensions inside the loop):

result = {}
for key, items in groupby(rows, key=lambda r: r[0]):
    result[key] = items   # stores the ITERATOR, not the data

# Later, outside the loop, the iterators are spent:
for key, items in result.items():
    print(key, list(items))   # b []   a []   <- empty!

Right — materialize the group immediately with list():

result = {}
for key, items in groupby(rows, key=lambda r: r[0]):
    result[key] = list(items)  # snapshot now, while valid

print(result)  # {'a': [('a', 2), ('a', 4)], 'b': [('b', 1), ('b', 3)]}

The correct, idiomatic pattern

Sort by the key, then snapshot each group with list() (or any consuming call) before moving on:

from itertools import groupby

rows = [("b", 1), ("a", 2), ("b", 3), ("a", 4)]
rows.sort(key=lambda r: r[0])

by_level = {
    level: list(items)
    for level, items in groupby(rows, key=lambda r: r[0])
}

print(by_level["a"])  # [('a', 2), ('a', 4)]

This streams the input (constant memory for the grouping step) and gives you real, reusable data.

groupby vs defaultdict — pick the right tool

  • Use groupby when input is (or can be) sorted and you want a streaming, one-pass grouping — great for log files, CSV rows, or huge streams.
  • Use collections.defaultdict(list) when order doesn’t matter and you’d rather not sort first. It handles unsorted data trivially:
from collections import defaultdict

buckets = defaultdict(list)
for r in [("b", 1), ("a", 2), ("b", 3), ("a", 4)]:
    buckets[r[0]].append(r)
# works without sorting, but loads everything into memory

Takeaways

  • groupby groups only consecutive equal keys — sort by the key first.
  • Group iterators are lazy and one-shot — consume them (usually with list()) before the outer loop advances.
  • Need unsorted grouping or must keep everything in memory? defaultdict(list) is often simpler.

Master these two traps and groupby becomes one of the cleanest tools in your data-processing toolkit.