Iterables and iterators

You have written hundreds of for loops over lists, strings, dictionaries and files. They all look the same from the outside, yet a list keeps its items in memory while a file reads one line at a time from disk. What lets one for statement work with all of them?

The answer is a small agreement called the iteration protocol. Once you understand it you can make your own objects loopable, read the first line of a data stream and process the rest separately, and avoid one of the most confusing bugs in Python: data that silently "disappears" the second time you loop over it. Everything in the next two lessons — generators and pipelines — is built on this protocol.

Two roles: iterable and iterator

• An iterable is anything you can loop over: a list, tuple, string, dict, set, range, file. Technically, it is an object whose __iter__ method returns an iterator. You ask for one with the built-in iter(obj).
• An iterator is the object that does the walking. It remembers its position and hands out the next item each time you call next(it) (which calls its __next__ method). When there are no items left it raises StopIteration.

A list is iterable but is not itself an iterator: you can loop over the same list many times because each loop asks for a fresh iterator. An iterator is single-use: once it has handed out every item, it is exhausted and stays empty.

orders = ["A-101", "A-102", "A-103"]

it = iter(orders)      # ask the list for an iterator
print(type(it).__name__)
print(next(it))        # A-101
print(next(it))        # A-102
print(next(it))        # A-103
print(next(it, "no more orders"))  # a default instead of StopIteration

What a for loop really does

This loop:

for order in orders:
    print(order)

is roughly equivalent to:

it = iter(orders)
while True:
    try:
        order = next(it)
    except StopIteration:
        break
    print(order)

The for statement calls iter() once, calls next() repeatedly, and treats StopIteration as the normal signal to stop — not as an error. sum(), list(), max(), sorted(), in, unpacking and comprehensions all use the same protocol.

readings = iter([12, 15, 11])

for r in readings:
    print("first loop:", r)

for r in readings:
    print("second loop:", r)

print("done")

Using next() on purpose

Sometimes you want to treat the first item differently — a CSV header, a title line, a starting balance. Get an iterator, take the first item with next(), then loop over the rest. The loop continues from where next() left off.

lines = ["sku,qty", "PEN-01,40", "INK-07,12", "PAD-33,5"]

rows = iter(lines)
header = next(rows)
print("columns:", header.split(","))
for line in rows:          # continues after the header
    sku, qty = line.split(",")
    print(sku, int(qty))

Writing an iterator class

To make your own iterator, give a class two methods:

• __iter__(self) returns the iterator — for an iterator class, that is self.
• __next__(self) returns the next value, or raises StopIteration when finished.

Here is an iterator over the invoice numbers in a range, formatted as text:

class InvoiceNumbers:
    def __init__(self, first, last):
        self.current = first
        self.last = last

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.last:
            raise StopIteration
        number = f"INV-{self.current:04d}"
        self.current += 1
        return number


for n in InvoiceNumbers(98, 101):
    print(n)

numbers = InvoiceNumbers(1, 2)
print(list(numbers))
print(list(numbers))   # exhausted: an iterator is single-use

Iterable versus iterator in your own classes

If you want an object that can be looped over many times — like a list — make it an iterable: its __iter__ returns a new iterator each time instead of self. A quick way is to return an iterator over some internal data (return iter(self.items)); in the next lesson you'll see that a generator is the neatest way.

When not to write an iterator class: most of the time you don't need one. If the data already fits in a list, return the list. If you need lazy, one-at-a-time values, a generator function (next lesson) does the same job in a few lines. Write a class when the iterator needs extra methods or state other code inspects.

split_header(["sku,qty", "PEN-01,40", "INK-07,12"])
# ('sku,qty', ['PEN-01,40', 'INK-07,12'])
split_header([])
# (None, [])

list(Countdown(3))   # [3, 2, 1]
list(Countdown(0))   # []
c = Countdown(2)
next(c), next(c)     # (2, 1)
next(c)              # raises StopIteration

rota = ShiftRota(["Huda", "Omar"], 3)
list(rota)   # [(1, 'Huda'), (2, 'Omar'), (3, 'Huda')]
list(rota)   # the same list again

summarize([20.0, 22.0, 24.0])         # (3, 22.0)
summarize(iter([20.0, 22.0, 24.0]))   # (3, 22.0)  -- currently (3, 0.0)
summarize(iter([]))                   # (0, None)