Generators in Python Explained: Lazy, Powerful, and Surprisingly Fun

Introduction #

Generators are a core part of Python’s toolkit for writing efficient, readable code—especially when working with large data sets or streams of information. They allow you to produce values on the fly, rather than creating and storing entire collections in memory. This is known as lazy evaluation, and it’s one of the key benefits of using generators.

If you’ve used for loops, you’ve already been working with iterators. And if you’ve seen the yield keyword in Python, you’ve had a brush with generators—even if you weren’t quite sure what was happening under the hood.

Whether you're new to generators or looking to understand them more deeply, this guide is designed to be clear, thorough, and practical. By the end, you’ll know how generators work, how to use them effectively, and how they can help you write more scalable Python code.

By the way, here's the video version of this article. Perfect if you want a short, high level overview before diving into this more in depth article.

Iterators and Iterables: The Foundation of Generators #

Before we jump into generators, we need to take a short detour into the land of iterators and iterables — because generators don’t just appear out of thin air. They’re built on top of Python’s iterator protocol, and understanding that is the key to unlocking how generators really work.

Want a deeper dive into iterators and iterables? Check out our complete guide to Python iterators vs iterables for a thorough explanation of these fundamental concepts.

What’s an Iterable, Anyway? #

In Python, an iterable is any object you can loop over with a for loop. Lists, tuples, strings, sets—they all qualify. If you can say for item in something, that something is an iterable.

Under the hood, Python calls the __iter__() method on the object, which returns an iterator.

numbers = [1, 2, 3]
iterator = iter(numbers)

And What’s an Iterator? #

An iterator is an object that knows how to give you one item at a time, until there are no more items left. It must implement two methods:

__iter__() – which returns the iterator object itself.
__next__() – which returns the next item, or raises StopIteration when it’s done.

numbers = [1, 2, 3]
iterator = iter(numbers)

print(next(iterator))  # 1
print(next(iterator))  # 2
print(next(iterator))  # 3
print(next(iterator))  # Boom! StopIteration

Where Do Generators Fit Into This? #

Generators are a shortcut. They’re Python’s way of saying, “Why write a full class with __iter__() and __next__() just to make an iterator? Here, use this instead.”

When you write a function with yield, Python automatically creates an iterator for you. You don’t need to define any special methods. The generator function returns a generator object that follows the iterator protocol. It’s like getting all the benefits of an iterator without the boilerplate code.

So, in summary:

Iterables are things you can loop over.
Iterators are the engines doing the actual looping.
Generators are a slick, Pythonic way to build your own iterators with far less hassle.

Generator Functions and the Mysterious yield #

If you’ve ever written a function in Python, you’re already familiar with return. It hands back a value and ends the function’s life then and there. That’s it. Goodbye, farewell, thanks for the data.

But yield? yield is different.

When a function uses yield, it doesn’t end. It pauses.

This is the moment you realize generator functions aren’t your standard functions — they’re functions that remember. They pause execution, save their state, and resume exactly where they left off the next time you ask for a value. Like bookmarks for code.

Writing Your First Generator Function #

Here’s a simple example:

def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

Calling this function won’t run it immediately. Instead, it returns a generator object:

counter = count_up_to(3)
print(next(counter))  # 1
print(next(counter))  # 2
print(next(counter))  # 3
print(next(counter))  # StopIteration

Each time you call next(), it picks up where it left off. When there’s nothing left to yield, Python raises StopIteration to signal the end.

yield vs return #

Let’s be clear: yield doesn’t just return a value — it suspends the function. The next time it’s resumed, it picks up after the yield statement, with all variables and local state intact.

If return is a period, yield is more like a semicolon. The sentence isn’t over.

And yes, you can use return in a generator — but if you do, it signals the end of the generator. Python will treat it as a polite way of saying, “I’m done now,” and stop the iteration.

What Does a Generator Function Actually Return? #

It returns a generator object. This object:

Implements the iterator protocol (__iter__() and __next__())
Can be used in a for loop
Can be passed around like any other object

You can even peek inside it with a few introspective tricks:

gen = count_up_to(5)
print(gen.__iter__() is gen)  # True – it's an iterator

This object is what makes generators so powerful. You don’t run the function — you get a recipe for running it one step at a time.

Generator Functions vs Regular Functions: yield vs return #

At first glance, generator functions look like regular functions. They have parameters, indentation, maybe a loop or two. But throw in a yield, and suddenly the rules change.

Let’s break down how these two types of functions differ—and why that matters.

Regular Functions: One Shot, One Return
Regular functions do one thing: run their code and hand back a result with return.

Regular Functions: One Shot, One Return #

Regular functions do one thing: run their code and hand back a result with return.

def get_numbers():
    return [1, 2, 3]

You call the function, you get the list. That’s it. The function does its job, then vanishes into the void. Done.

Generator Functions: Pause and Resume #

Now watch what happens when we use yield instead:

def get_numbers():
    yield 1
    yield 2
    yield 3

Calling this function doesn't execute the body. It returns a generator object, which you can then step through one value at a time.

gen = get_numbers()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3

Instead of returning a full result, the function pauses at each yield, hands back a single value, and remembers where it left off. This statefulness is what gives generators their magic.

Function Flow: A Tale of Two Behaviors #

With return, you exit the function and lose all context. With yield, you suspend it — like a TV show on pause. All local variables, loop counters, and stack frames are preserved until the next next() call resumes execution.

This allows generator functions to:

Produce a sequence of values over time
Maintain internal state without external helpers
Avoid holding everything in memory at once

You Can Use Both (But Choose Wisely) #

Yes, you can technically use both yield and return in the same function, but with a caveat: return ends the generator early. If you include a return value, it becomes part of the StopIteration exception — but that’s usually only useful in advanced cases (we’ll get there).

def countdown(n):
    while n > 0:
        yield n
        n -= 1
    return "Blast off"  # This is hidden unless you catch StopIteration

So use yield to emit values. Use return if you need to stop early or signal a final message behind the scenes.

Generator Expressions #

By now, you’ve seen how generator functions use yield to produce values one at a time. But Python, being Python, offers an even more concise way to create generators — without defining a full function.

Enter: generator expressions.

The Sibling of List Comprehensions #

If you’re familiar with list comprehensions, generator expressions will look instantly familiar. The only real difference? Parentheses instead of square brackets.

# List comprehension
squares_list = [x * x for x in range(5)]

# Generator expression
squares_gen = (x * x for x in range(5))

The list comprehension creates all values right away and stores them in memory. The generator expression produces them one at a time, on demand.

In other words:

The list version is eager.
The generator version is lazy—in a good way.

When to Use Generator Expressions #

They’re perfect when:

You’re dealing with large data streams
You only need to loop through the results once
You want to save memory and avoid unnecessary allocations

For example, summing a large range of numbers:

total = sum(x * x for x in range(10_000_000))

There’s no need to store 10 million values in memory. The generator expression hands them to sum() one at a time — no memory bloat, no fuss.

So... Are Generator Expressions Always Better? #

Not always. If you plan to iterate over the data multiple times, or need random access, a list is more appropriate. Generator expressions are one-pass, fire-and-forget.

You also can't index or slice them:

gen = (x for x in range(10))
print(gen[0])  # TypeError: 'generator' object is not subscriptable

But if your goal is to loop once through a large or expensive-to-compute dataset? Generator expressions are a clean, readable win.

Consuming Generators: next(), for Loops, and the Quiet End #

So you’ve created a generator — either from a function with yield or a slick one-liner with parentheses. Now what?

You consume it.

Generators don’t produce values until you ask. And there are two primary ways to do the asking: next() and for loops.

The Manual Way: next() #

Calling next() on a generator gives you the next value—just one at a time:

def greet():
    yield "Hello"
    yield "Bonjour"
    yield "Hola"

g = greet()

print(next(g))  # Hello
print(next(g))  # Bonjour
print(next(g))  # Hola

Once the generator has nothing left to give, it raises a StopIteration exception. Python’s polite way of saying, “We’re done here.”

print(next(g))  # StopIteration

If you’re using next() directly, it’s a good idea to use try/except or the second argument to next() to handle the end gracefully:

print(next(g, "No more greetings"))  # No more greetings

The Pythonic Way: for Loops #

Most of the time, you won’t use next() directly. You’ll use a for loop, which handles the heavy lifting for you:

for greeting in greet():
    print(greeting)

Under the hood, the for loop:

Calls iter() on the object (which returns the generator itself)
Repeatedly calls next() on it
Stops when it hits StopIteration

And it does all this without bothering you with exceptions or stack traces. Efficient and well-mannered.

You Can Only Consume Generators Once #

Generators don’t “rewind”. Once they’re exhausted, they’re done:

gen = (x for x in range(3))

for num in gen:
    print(num)

for num in gen:
    print(num)  # Nothing happens here

If you need to iterate again, you’ll need to recreate the generator.

Memory and Performance: Generators vs List Comprehensions #

You’ve probably heard that generators are more memory - efficient than lists. But what does that actually mean? Is it just tech folklore, or can we back it up with numbers?

Let’s investigate.

The Big Difference: Eager vs Lazy #

Let’s start simple:

squares_list = [x * x for x in range(1_000_000)]
squares_gen = (x * x for x in range(1_000_000))

Both look similar. One uses brackets, the other uses parentheses. But behind the scenes:

squares_list builds all the numbers in memory, right away.
squares_gen builds them one at a time, only when you ask.

Let’s check their memory usage:

import sys

print(sys.getsizeof(squares_list))  # Large – stores all values
print(sys.getsizeof(squares_gen))   # Small – just the generator object

Since the generator doesn’t store all those numbers in memory, it uses very little memory. It stores just enough to know how to produce the next item.

And What About Speed? #

There’s a common myth that generators are slower than lists. And while that can be true in some contexts, it's not a rule.

Try this comparison:

import time

start = time.time()
sum([x * x for x in range(50_000_000)])
print("List:", time.time() - start)

start = time.time()
sum(x * x for x in range(50_000_000))
print("Generator:", time.time() - start)

You might expect the list to be faster—but often, the generator is slightly quicker. Why?

Because:

The generator avoids building a giant list in memory.
Functions like sum() are optimized for consuming iterables.
The generator skips the overhead of allocating and populating a massive list.

That said, if you need to iterate multiple times, or you need random access to the results, the list still wins. It's built and ready to go, and you can loop it all day long.

So Which Should You Use? #

It comes down to intent:

Need all the data at once, or multiple passes? Use a list comprehension.
One-pass, streaming-style logic? Generator expressions are your friend.
Working with gigantic datasets? Always think twice before allocating a list.

Here’s a summary:

Feature	List Comprehension	Generator Expression
Memory Usage	High (stores everything)	Low (generates on demand)
Speed (1-pass)	Comparable or slower	Often slightly faster
Speed (multi-pass)	Faster (data already built)	Not reusable
Reusable	Yes	No
Supports Indexing	Yes	Nope

yield from: Delegation Without the Boilerplate #

Sometimes, your generator function wants to hand off responsibility — like a manager who finally decides to delegate. That’s where yield from comes in. It's Python’s way of saying, “Let this other iterable take it from here.”

The Manual Way: Yielding in a Loop #

Let’s say you want to yield every value from a list or another generator:

def numbers():
    yield 1
    yield 2
    yield 3

def wrapper():
    for value in numbers():
        yield value

This works. It’s fine. But it’s also boilerplate — every time you want to yield from a sub-generator or iterable, you’re writing a loop.

The Clean Way: yield from #

Here’s the same example, cleaner and more expressive:

def wrapper():
    yield from numbers()

This does the exact same thing — but Python handles the looping for you.

In fact, yield from doesn’t just yield values — it also:

Passes exceptions into the sub-generator
Returns the final value from the sub-generator (if it uses return)
Handles .send(), .throw(), and .close() transparently (more on those later)

It’s like generator inheritance.

Yielding from Anything Iterable #

You’re not limited to other generators. yield from works with any iterable — lists, tuples, sets, even strings:

def letters():
    yield from "ABC"

for char in letters():
    print(char)  # A B C

It’s a clean, readable way to flatten nested generators or insert sub-steps into a pipeline without creating a tangle of for loops.

A Practical Use Case: Recursive Generators #

Let’s say you want to flatten a nested structure:

def flatten(items):
    for item in items:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

nested = [1, [2, [3, 4]], 5]

print(list(flatten(nested)))  # [1, 2, 3, 4, 5]

This kind of recursive delegation is where yield from really shines. Without it, you'd need nested loops and code that isn't as concise.

Advanced Generator Control: .send(), .throw(), and .close() #

By default, generators are polite little machines: you say next(), they give you the next value. But sometimes you want more control. Maybe you want to send in data, inject an error, or politely ask the generator to shut down.

Good news: generators allow you to do all of this with the help of .send(), .throw(), and .close().

.send(value): Talk to Your Generator #

Generators don’t just yield values — they can receive them too. To do this, you use .send().

Here’s a simple example:

def greeter():
    name = yield "What’s your name?"
    yield f"Hello, {name}!"

gen = greeter()
print(next(gen))         # "What’s your name?"
print(gen.send("Alice")) # "Hello, Alice!"

What’s happening here?

The first yield pauses the generator and gives you a prompt.
When you call .send("Alice"), the generator resumes, and "Alice" gets assigned to the variable name.

This lets you write coroutines — functions that pause, resume, and react to input.

⚠️ Note: You must start the generator with next() or send(None) before sending real values. Otherwise, Python won’t be ready to receive.

.throw(exc_type): Raise an Exception Inside a Generator #

Want to simulate an error inside a generator? You can inject an exception with .throw():

def controlled():
    try:
        yield "Running..."
    except ValueError:
        yield "ValueError caught!"
    yield "Done"

gen = controlled()
print(next(gen))                 # "Running..."
print(gen.throw(ValueError))     # "ValueError caught!"
print(next(gen))                 # "Done"

You don’t have to actually raise errors from inside the generator — you can send them in from the outside.

This is useful in coroutine frameworks and when coordinating complex stateful flows.

.close(): Shut It Down #

When you’re done with a generator and want to clean up, you can call .close(). This raises a GeneratorExit inside the generator, giving it a chance to tidy up before shutting down:

def tidy_up():
    try:
        while True:
            yield "Working..."
    finally:
        print("Cleaning up!")

gen = tidy_up()
print(next(gen))  # "Working..."
gen.close()       # "Cleaning up!"

This is especially handy when your generator is managing resources (like open files or network connections) and needs to release them when done.

When to Use Generators (and When Not To) #

At this point, you might be thinking: “Generators are amazing. I should use them everywhere!”

Hold that thought. While generators are powerful and elegant, they’re not always the right tool. Let’s look at some use cases where generators shine — and a few where they really don’t.

✅ Use Generators When… #

You’re Working with Large or Infinite Data

Generators don’t load everything into memory. That makes them perfect for huge files, data streams, or even infinite sequences:

def count_forever(start=0):
    while True:
        yield start
        start += 1

Reading lines from a log file, streaming API data, or crunching numbers without drowning your RAM? Generators are ideal.

You Only Need One Pass Through the Data

Generators are disposable: once they’re done, they’re done. If you just need to loop once—say, for filtering or aggregation—they’re a great fit:

total = sum(x for x in range(1000000) if x % 2 == 0)

One pass. Minimal memory. Job done.

You Want Lazy Evaluation

Sometimes you don’t know how much data you’ll need — or when. Generators let you delay the work until it’s absolutely needed, which can lead to faster startup times and lower resource usage.

You’re Modeling a Pipeline

Generators fit beautifully into a pipeline-style architecture where data flows through stages of transformation:

def read_lines(file):
    for line in file:
        yield line.strip()

def filter_lines(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

with open("log.txt") as f:
    for error in filter_lines(read_lines(f)):
        print(error)

Each function does one thing. Clean, efficient, and scalable.

❌ Avoid Generators When… #

You Need Random Access

Generators don’t support indexing or slicing. Once a value is yielded, it’s gone. If you need to go back and forth or access specific items, use a list.

gen = (x for x in range(10))
# gen[2]  ❌ This will raise an error.

You Need to Reuse the Data

Generators are one-time use. If you iterate through it once, it’s exhausted. You’ll need to re-create the generator if you want to iterate again.

You’re Prioritizing Speed on Small Datasets

While generators are often fast, list comprehensions can sometimes beat them on small or medium-sized inputs — especially when reused. Python has spent years optimizing lists. For short, hot code paths, the extra overhead of generator machinery might not be worth it.

Your Logic Is Too Complex

Generators can be elegant — but they can also turn into spaghetti if you overload them with state, side-effects, and control flow. If your generator function starts needing a diagram to follow, consider using a class or refactoring.

Bottom Line #

Generators are a fantastic tool in your Python toolkit — but like all tools, they have their ideal jobs. Use them when:

Memory matters
Laziness is an advantage
You want clean, composable iteration

Use lists (or other containers) when:

You need flexibility
You need to keep the data around
You need to access it more than once

As always, the best Python code is the one that makes your intent obvious — to both the interpreter and your future self.

Exploring Generators in Python — From Basics to Advanced