Python Iterators and Iterables: A Hands-On Guide

Introduction #

When you're writing Python, you’re probably using iterables and iterators more often than you realize. Any time you loop through a list, unpack a tuple, or read lines from a file, you’re leaning on Python’s iteration machinery.

It’s one of those behind-the-scenes features that just works—until you start asking how and why. Then things get a bit more interesting.

This post is here to walk you through the logic and mechanics behind iteration in Python. We'll look at the difference between iterables and iterators, what responsibilities an iterator actually has, and how Python’s iteration protocol makes all this work seamlessly.

By the end, you’ll not only know how to use iterators—you’ll know how to build your own. And once you understand how Python handles iteration under the hood, a lot of things in the language start to click into place.

---

By the way, here's the video version of this article. Perfect if you want a short, high level overview before diving into this more in depth article.

---

What Is Iteration #

Before we dive into the technical details, let’s start with the big picture. What does it actually mean to iterate over something in Python?

In plain terms: iteration is the process of going through items in a collection, one at a time, in a specific order.

That’s it. That’s the concept.

You’ve probably done this countless times:

for number in [1, 2, 3]:
    print(number)

Behind that unassuming for loop is a lot of clever design. Python doesn't just "know" how to loop over things — it relies on a specific set of rules and objects to make it happen. And that’s where iterables and iterators come in.

When Do You Need Iteration? #

Iteration shows up in more places than just for loops. You’re using it when you:

• Loop over elements in a list or tuple
• Read lines from a file
• Use a generator expression
• Unpack values (a, b = some_tuple)
• Call join() on a string with a list of items
• Even when using comprehensions like [x for x in something]

In short: if you’re processing items one-by-one, there’s iteration happening. And understanding the mechanics gives you more control, more efficiency, and fewer surprises when things don’t behave as expected.

Iterables vs. Iterators (At a Glance) #

Let’s set the stage with a high-level distinction. You’ll get the full breakdown later, but here’s the key idea:

An iterable is something you can iterate over. Lists, tuples, strings, sets—they’re all iterables. But they don’t do the iteration themselves.

An iterator is the thing that actually does the work. It keeps track of where you are and knows how to get the next item.

You can think of an iterable as a recipe, and an iterator as the chef who follows it. You can give the same recipe to multiple chefs (iterators), but each chef works through the steps independently.

Here’s a tiny preview:

my_list = [10, 20, 30]

iterator = iter(my_list)
print(next(iterator))  # 10
print(next(iterator))  # 20
print(next(iterator))  # 30
print(next(iterator))  # raises StopIteration

Calling iter(my_list) turns the list into an iterator — something with a __next__() method that knows how to give you the next item and raise a StopIteration when it's done.

The Responsibilities of an Iterator #

Now that we’ve got a general sense of what iterables and iterators are, let’s zero in on the iterator itself — the behind-the-scenes worker in the iteration process.

So what exactly does an iterator do?

In short, an iterator has two main jobs:

• Return one item at a time from a sequence or stream
• Keep track of where it is in the sequence, so it knows what comes next

Unlike an iterable (which is basically just a collection of data), an iterator adds a bit of state and behavior into the mix. It’s not just a container — it’s a little machine with a memory.

Two Magic Methods: __iter__() and __next__() #

To qualify as an iterator in Python, an object needs to implement two special methods:

class SomeIterator:
    def __iter__(self):
        return self  # Yep, iterators return themselves

    def __next__(self):
        # Logic to return the next item (or raise StopIteration)
        pass

Let’s break this down:

• __iter__() is what makes the object iterable. It tells Python, “Yes, I know how to produce values one at a time.”
• __next__() is where the actual work happens. Each call to next() triggers this method to produce the next value (and to raise StopIteration when it’s out of items).

Why Does an Iterator Return Itself from __iter__()? #

This may feel a little circular:

iter_obj = iter(some_iterator)

But wait… isn’t some_iterator already an iterator?

Exactly. When you call iter() on an iterator, it just returns itself. This is part of how Python’s for loop works internally:

iterator = iter(iterable)
while True:
    try:
        item = next(iterator)
        # Do something with item
    except StopIteration:
        break

By ensuring that __iter__() returns self, the object can be used directly in any context that expects an iterable.

This also explains why trying to next() over a plain list doesn't work — you need to convert it to an iterator first.

numbers = [1, 2, 3]

next(numbers)          # ❌ TypeError: 'list' object is not an iterator
next(iter(numbers))    # ✅ Returns 1

What About StopIteration? #

The moment an iterator runs out of items, it raises a StopIteration exception. This is Python’s elegant way of saying, “We’re done here.”

This is also how the for loop knows when to stop. It keeps calling next() until it hits that exception, at which point it quietly exits.

You don’t usually see this in your own code (unless you’re using next() manually), but it’s an important part of the protocol.

Building a Custom Iterator: CustomRange #

Enough theory — let’s put it into practice.

To really understand how iterators work, it helps to build one yourself. So let’s roll up our sleeves and recreate a simplified version of Python’s built-in range() function. We’ll call it CustomRange.

What Should CustomRange Do? #

Like range(start, end), our class should:

• Take a start and end value
• Return numbers from start up to (but not including) end
• Work inside a for loop
• Support manual iteration using next()

Create the Class #

Here’s a basic version of CustomRange:

class CustomRange:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.end:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

Let’s break it down:

• The constructor __init__ sets up the internal state — where to start and where to stop.
• __iter__() returns the iterator object itself (required).
• __next__() does the actual work of returning the current value and moving the pointer forward.

Using CustomRange in a for Loop #

Thanks to Python’s iteration protocol, your class now works in a loop:

for num in CustomRange(3, 7):
    print(num)

# 3
# 4
# 5
# 6

Notice how clean this is — Python calls iter() under the hood, then repeatedly calls next() until it hits StopIteration.

Manual Iteration with next() #

You can also drive this manually, just to see what’s going on:

r = CustomRange(1, 4)
print(next(r))  # 1
print(next(r))  # 2
print(next(r))  # 3
print(next(r))  # Raises StopIteration

This helps you visualize how the iterator maintains its own internal state (self.current) between calls.

What Happens After StopIteration? #

Once StopIteration is raised, the iterator is done. You can’t “reset” it by calling next() again — you’ll keep getting that exception.

If you want to iterate again, you need a fresh instance:

for num in CustomRange(0, 3):
    print(num)  # Works

# But:
cr = CustomRange(0, 2)
list(cr)       # [0, 1]
list(cr)       # [] ← it's already been exhausted!

This illustrates one of the key traits of iterators: they’re single-use.

Another Custom Iterator: VowelIterator #

Let’s make an iterator that returns just the vowels from a given word. This shows how iterators can be used not just to walk through numeric ranges, but to filter and yield items based on some rule.

class VowelIterator:
    def __init__(self, word):
        self.word = word
        self.index = 0
        self.vowels = "aeiouAEIOU"

    def __iter__(self):
        return self

    def __next__(self):
        while self.index < len(self.word):
            char = self.word[self.index]
            self.index += 1
            if char in self.vowels:
                return char
        raise StopIteration

Here’s what’s going on:

• We store the word and keep track of our position using self.index.
• On each call to __next__(), we check if the current character is a vowel.
• If it is, we return it. If not, we keep going.
• Once we’ve gone through the entire word, we raise StopIteration.

Let's try it out:

for ch in VowelIterator("iterator"):
    print(ch)

# i
# e
# a
# i
# o

Why This Matters #

This iterator highlights a few key ideas:

• Iterators can filter values as they go — they don’t have to return everything.
• You have full control over how and when data is yielded.
• You can use iterators for non-numeric sequences, like strings, lists, or even lines from a file.

Inheriting from collections.abc.Iterator #

So far, we’ve built iterators using the raw protocol: implement __iter__() and __next__(), and you’re good to go.

But Python being Python, there's a formal way to say, "Hey, this class is an iterator." That’s where collections.abc.Iterator comes in.

What Is collections.abc.Iterator? #

The collections.abc module contains abstract base classes for many core Python interfaces — including iterators.

If you inherit from collections.abc.Iterator, your class must implement:

• __iter__() (which should return self)
• __next__() (which does the actual data fetching)

It’s not required, but it’s great for:

• Clarity: Signals intent that this is a proper iterator
• Validation: Makes sure you’re not accidentally skipping something important
• Type checking: Works nicely with tools like mypy

Let’s Update VowelIterator #

Here’s how we’d update our previous example to inherit from the base class:

from collections.abc import Iterator

class VowelIterator(Iterator):
    def __init__(self, word):
        self.word = word
        self.index = 0
        self.vowels = "aeiouAEIOU"

    def __next__(self):
        while self.index < len(self.word):
            char = self.word[self.index]
            self.index += 1
            if char in self.vowels:
                return char
        raise StopIteration

Notice:

• We’ve removed __iter__() entirely. That’s because the base Iterator class already defines it to return self — so we don’t have to.
• We now have a self-validating, future-proof iterator that will fail loudly if we forget __next__().

Should You Always Do This? #

For quick one-offs or examples in a tutorial (like this one), skipping the base class is fine.

But in a larger codebase — especially if you're sharing your code with others — inheriting from collections.abc.Iterator is a clean, explicit way to show your class follows Python's iterator protocol.

__iter__() vs __next__(): Who Does What? #

Let’s pause and take a proper look at the two most important methods in this whole iteration dance: __iter__() and __next__(). If these two were coworkers, __iter__() would be the person handing out the to-do list, and __next__() would be the one actually doing the tasks — until they burn out and raise a StopIteration.

The Iterable-Iterator Relationship #

Here’s the basic division of labor:

• An iterable is any object that has a __iter__() method. It can be looped over (e.g., in a for loop).
• An iterator is an object that has both:
  • __iter__() — which returns itself
  • __next__() — which returns the next item (or raises StopIteration when done)

Let’s put that into perspective with a list:

my_list = [10, 20, 30]

This is an iterable. It has a __iter__() method, so you can loop over it.

But:

next(my_list)

That’ll raise:

TypeError: 'list' object is not an iterator

Turning an Iterable into an Iterator #

You can convert an iterable into an iterator using the built-in iter() function:

iterator = iter(my_list)
print(next(iterator))  # 10
print(next(iterator))  # 20

Now this iterator has both __iter__() and __next__(). It’s the real deal.

Why Do Iterators Have __iter__()? #

Good question. At first glance, it feels a bit... redundant, right?

But this is what allows Python to treat both iterables and iterators the same in for loops. The loop machinery always does this:

it = iter(some_object)
while True:
    try:
        item = next(it)
    except StopIteration:
        break
    # do something with item

If some_object is already an iterator, iter(some_object) just returns it. So __iter__() on an iterator simply returns self.

This keeps things simple and consistent — any object passed to iter() will behave as expected, whether it's an iterable or an iterator.

Simulating a for Loop with while and next() #

We’ve talked about how iterables can be looped over, but what does Python actually do when you write a for loop?

Spoiler: it’s not magic. It’s just a while loop with next() under the hood.

Here’s an example using our custom VowelIterator class from earlier:

vowels = VowelIterator("iteration")
while True:
    try:
        char = next(vowels)
        print(char)
    except StopIteration:
        break

# i
# e
# a
# i
# o

Let’s break this down:

• iter() turns the list into an iterator.
• next() retrieves the next item.

If the iterator is exhausted, a StopIteration exception is raised, and the loop exits.

As you can see, it works exactly the same as a stock standard for loop:

for ch in VowelIterator("iterator"):
    print(ch)

# i
# e
# a
# i
# o

Why Does This Matter? #

Understanding how next() and StopIteration work gives you:

• More control: You can decide exactly how to iterate and when to stop.
• More insight: Debugging or customizing iteration behavior becomes easier.
• More appreciation: For the humble for loop, which quietly does all of this for you.

What Iterators Can’t Do #

Iterators may seem like the Swiss Army knife of looping, but they're not without their caveats. They're lean, they're lazy, and they go in one direction — but ask them to jump around or restart, and they'll look at you like you’ve asked a cat to fetch.

Let’s look at what you can’t do with iterators (and why it matters).

No Indexing #

Unlike lists or tuples, you can’t do this:

it = iter([10, 20, 30])
print(it[0])  # ❌ TypeError: 'list_iterator' object is not subscriptable

That’s because iterators don’t support random access. They only remember the current position, not the entire sequence. Once something is passed, it’s gone.

No Rewinding #

Once an iterator has been consumed, it’s done.

it = iter([1, 2, 3])
print(next(it))  # 1
print(next(it))  # 2
print(next(it))  # 3
print(next(it))  # ❌ StopIteration

Want to start over? You’ll need a new iterator.

They Don't Store History #

An iterator doesn’t remember what it’s already returned. If you need that data, you’ll have to store it yourself:

data = [5, 6, 7]
it = iter(data)
seen = []

for item in it:
    seen.append(item)

# 'seen' now has the history, but 'it' is exhausted

Once exhausted, an iterator stays that way. It doesn't reset automatically.

They Don’t Know Their Own Length #

This doesn’t work:

it = iter([1, 2, 3])
len(it)  # ❌ TypeError: object of type 'list_iterator' has no len()

Many iterators don’t know how long they are — especially when the data they represent is coming from a stream, generator, or large file. That’s part of what makes them efficient.

What Iterators Can't Do Makes Them Efficient #

These limitations aren’t bugs — they’re features. Iterators are designed for efficiency, especially when dealing with large or infinite data streams.

Because they:

• Only produce one value at a time
• Don’t store previous values
• Can represent data sources that have no clear end (like a sensor or socket stream)

…they use less memory and can handle much bigger data than a list could.

Real-World Use Cases for next() #

I’ve sung the praises of for loops — they’re clean, readable, and handle all the iteration housekeeping for you. But sometimes, reaching for next() directly gives you more control. Let’s look at a practical use case.

Skipping the Header Row in a CSV File #

When reading from a CSV file, you often want to skip the header row before processing the rest. With a for loop, this requires a little extra logic. But with next(), you can skip straight past it.

import csv

with open("data.csv") as file:
    reader = csv.reader(file)
    next(reader)  # Skip the header row

    for row in reader:
        print(row)

Here, reader is an iterator. Calling next(reader) consumes just the first row — no fuss, no flags, no counters.

When to Use next() Over a for Loop #

Use next() when you need:

• To manually control the flow of iteration
• To skip or inspect specific items before entering a loop
• To peek at the next item (with a bit of exception handling)
• To integrate iteration into custom control structures (e.g., parsers or data streams)

Use a for loop when you:

• Just need to walk through every item
• Want cleaner, more Pythonic code
• Don’t need that extra control

Don’t Forget: next() Can Raise StopIteration #

If you call next() too many times, you'll eventually hit the end of the iterator when a StopIteration exception is raised.

To avoid running into a StopIteration exception unexpectedly, you can give next() a default:

next(nums, "No more items")  # returns the default instead of raising StopIteration

Memory Efficiency #

When it comes to managing memory, iterators are stealthy — they don’t take up space they don’t need, they operate one step at a time, and they never stick around longer than necessary. This makes them a perfect fit for dealing with large datasets or infinite sequences without clogging up your system's RAM.

Lists vs Iterators: A Memory Showdown #

Let’s say you want to process a million numbers. You could create a list:

import sys
nums = [n for n in range(1_000_000)]
print(sys.getsizeof(nums))  # 8448728 bytes - about 8MB

That list exists in memory — all one million items, just sitting there.

Now let’s do the same with an iterator:

import sys
nums = iter(range(1_000_000))
print(sys.getsizeof(nums))  # 40 bytes

In this case, memory usage stays tiny, because only one item at a time is pulled from the underlying sequence. You can loop through all of them without ever holding the entire range in memory.

Iterators are lazy which means values aren’t computed or stored until you actually need them. This laziness is the secret to their memory efficiency.

When Memory Efficiency Matters #

You’ll benefit from iterators when:

• Dealing with large files (like reading line by line)
• Processing real-time data streams (like logs or sensor data)
• Working with infinite sequences (yes, those exist!)
• Performing pipeline-style data transformations (a.k.a. chaining)

Here’s an example of reading a file one line at a time:

with open("big_file.txt") as f:
    for line in f:
        process(line)  # one line at a time!

The file object is an iterator — so this approach keeps memory usage minimal, no matter how many lines the file has.

Iterables vs Iterators #

By now, you’ve seen iterables and iterators in action, but let’s pause and zoom out. What really separates them? What do they have in common? When should you use one over the other?

Feature	Iterable	Iterator
Can be used in a for loop	✅	✅
Has __iter__() method	✅	✅
Has __next__() method	❌	✅
Can be passed to iter()	✅ (returns an iterator)	✅ (returns itself)
Remembers position during iteration	❌ (needs a new iterator each time)	✅
Supports indexing ([])	Often yes (e.g., lists, strings)	❌
Can be reused	✅ (e.g., re-looping a list)	❌ (exhausts after one pass)
Lazy (one item at a time)	❌	✅
Uses less memory	❌ (holds all items in memory)	✅

Iterable Unpacking in Python #

One of Python’s more elegant features is unpacking — the ability to assign multiple values from an iterable in a single line of code. It’s concise, readable, and works across a wide range of data types.

Basic Iterable Unpacking #

Let’s start with a simple example using a tuple:

point = (3, 7)
x, y = point
print(x)  # 3
print(y)  # 7

Because point is an iterable, Python creates an iterator behind the scenes and pulls values from it using next(). The number of variables on the left must match the number of items in the iterable — otherwise, you’ll get a ValueError.

The * Operator for Flexible Unpacking #

Python also supports a special * syntax for capturing multiple values:

first, *middle, last = [1, 2, 3, 4, 5]
print(first)   # 1
print(middle)  # [2, 3, 4]
print(last)    # 5

This is especially useful when you want to grab specific positions and don’t care about the rest — or when the number of elements varies but you only need a few key parts.

A Note on Iterator Exhaustion #

If you’re working with an actual iterator (as opposed to just an iterable), keep in mind that unpacking will consume it:

it = iter([10, 20, 30])
a, b, c = it
# `it` is now exhausted and can’t be reused

Once a value is retrieved from an iterator, it’s gone — unless you explicitly store the results somewhere.

Why This Matters #

Unpacking looks simple on the surface, but it’s a direct result of Python’s iterator protocol. Understanding what’s happening behind the scenes — how Python calls iter() and next() — can help you avoid surprises and write more predictable code.

Generators: Iterators, Simplified #

Now that you’ve seen how iterators work under the hood — with __iter__, __next__, and a fair bit of boilerplate — it’s time to introduce a tool that makes creating them much easier: generators.

Generators give you the same power as custom iterator classes, but with a lot less code and more readable logic.

The Connection Between Iterators and Generators #

At their core, generators are just iterators.

When you create a generator (either through a generator function or a generator expression), Python automatically implements the iterator protocol for you:

• The object returned has both __iter__() and __next__() methods.
• Each time you call next(), it resumes execution right where it left off.
• When the function completes, it raises StopIteration automatically.

In other words, generators are iterators — just with friendlier syntax.

Creating a Custom Range — With a Generator #

Let’s revisit our earlier CustomRange class, but this time implement it as a generator function:

def custom_range(start, end):
    current = start
    while current < end:
        yield current
        current += 1

Now you can use it just like before:

for num in custom_range(3, 7):
    print(num)

# 3
# 4
# 5
# 6

Under the hood, yield pauses the function and returns a value. On the next next() call, execution resumes right after the yield, with all local variables preserved.

Why Use Generators? #

Besides being less code to write, generators come with a few nice perks:

• Lazy evaluation: Values are produced one at a time, only when needed.
• Memory efficiency: No need to store the entire sequence in memory.
• Clean syntax: No need for a full class or managing state manually.

Compare that to the earlier class-based iterator — which required defining two special methods and maintaining state yourself — and it’s easy to see why generators are often the go-to approach when building custom iterators.

Generator Expressions #

Just like you can write list comprehensions, Python also supports generator expressions:

squares = (x * x for x in range(5))
print(next(squares))  # 0
print(next(squares))  # 1

This is another way to build lazy iterators — compact and Pythonic.

Generator Wrap-up #

If an object needs to behave like an iterator but doesn’t require complex state or behavior, use a generator. It’s the same result, with less ceremony.

Final Thoughts #

Understanding iterators and iterables isn't just about knowing what methods they have — it's about knowing how Python thinks about sequences, loops, and memory efficiency.

• Iterables are the data sources.
• Iterators are the tools for going through them, one step at a time.
• Generators are the Pythonic shortcut to making your own iterators.

Once you recognize the pattern, you’ll start seeing it everywhere — in loops, comprehensions, function calls, and more.