↓ Skip to main content
Antomor
  1. Blog/

Iterators & Generators in Python

·474 words·3 mins
Antonio Morrone
Author
Antonio Morrone
Staff/Tech Lead Software Engineer. Building software since 2013, in blockchain protocols & Web3 infrastructure since 2021. Security-minded. Remote from Italy.
Table of Contents

This post aims to describe the basic mechanisms behind iterators and generators.

Iterator protocol
#

As in many programming languages, Python allows you to iterate over a collection. The iteration mechanism is often useful when we need to scan a sequence, an operation that is very common in programming. In Python, the iterator protocol involves two components: an iterable and an iterator.

Iterable
#

The iterable is the container through which we want to iterate. It is the object that needs to be scanned to retrieve all the elements (or some of them). Some well-known iterables are lists, tuples, dictionaries, and ranges. In the iterator protocol, the iterable exposes an __iter__ method that returns an iterator object.

Iterator
#

The iterator is the data structure that allows scanning through the container. It could seem like a complication, but actually, with the separation of concerns, it lets the developer separate the concept of the container from the concept of iteration. The container object doesn’t need to keep the state of an iteration, and furthermore, on the same object, many iterations can take place at the same time, so keeping the iteration state in a different object is a must. The container is a collection of elements, while the iterator is a kind of handler for the container: it exposes the same elements (owned by the container) one by one, in a specific order. In the iterator protocol, the iterator exposes two methods: __iter__ and __next__. While the first one returns the object itself (which allows the use of both the container and the iterator in for and in statements), the latter returns the next item from the container. What makes the iterator end the iteration? The StopIteration exception.

Iterable and Iterator Examples
#

Below is an example of an iterator protocol implementation:

And its usage:

Generator
#

Generators are methods with yield statements. The yield statement has the power to suspend the function’s execution and store its state, so that it can be resumed. Behind the scenes, Python returns control to the function’s caller and saves the function’s state; this way, at the next execution, the function will start where it left off, without the developer needing to worry about the function’s state. Generators ARE iterators, but not vice versa.

Here is an example of a generator:

and its usage:

Async generator
#

Like generators, they are async functions with a yield statement.

Conclusions
#

  • Iterators are iterables.
  • Iterators are objects that implement the iterator protocol, consisting of implementing both __iter__ and __next__.
  • Iterator iterations stop when StopIteration is raised.
  • Generators are methods with yield statements.
  • Async generators are async methods with yield statements.
  • Whenever possible, generators should be the preferred method due to their simplicity, while the protocol implementation gives much more control.

Resources
#