Python yield Explained: Generators, Memory Efficiency, and Best Practices

Python's yield keyword can look like a variation of return, but it changes how a function runs. A function that yields values produces a generator: an iterator that can pause between values and resume its saved state when the next value is requested. This makes generators useful for sequential work with data that is large, streamed, or computed on demand.

What does yield do?

return gives a function's result back to its caller and ends that function call. yield produces one value and suspends the function. Its local variables and execution position are retained, and iteration can resume from that point.

A function containing yield is a generator function. Calling it creates a generator object; the function body starts running only when the generator is advanced.

return versus yield

Use a list when you want to build and return a small, finite collection all at once:

def numbers_with_return():
    return [1, 2, 3]

print(numbers_with_return())
# [1, 2, 3]

Use yield when values should be produced one at a time:

def numbers_with_yield():
    yield 1
    yield 2
    yield 3

for number in numbers_with_yield():
    print(number)

The loop requests each next value, so the caller can process it before asking for another. A generator is not a list, though: it supports iteration rather than general-purpose indexing and repeated traversal.

How a generator pauses and resumes

Each call to next() runs a generator until it reaches the next yield, returns that value, and pauses again. When there are no more yielded values, iteration ends with StopIteration; a for loop handles that signal automatically.

def steps():
    print("Start")
    yield 1
    print("Middle")
    yield 2
    print("End")
    yield 3

generator = steps()
print(next(generator))
print(next(generator))
print(next(generator))

Creating generator does not print anything. The first next() starts the body, prints “Start,” and returns 1. Later requests continue after the previous yield. This pause-and-resume behavior is why a generator can preserve state without constructing all its future values in advance.

Why generators can be memory efficient

A list comprehension constructs and retains every result:

numbers = [number for number in range(1_000_000)]

A generator expression computes values as they are requested:

numbers = (number for number in range(1_000_000))

for number in numbers:
    process(number)

Similarly, a generator function can describe a sequence without keeping a full result list:

def generate_numbers(limit):
    for number in range(limit):
        yield number

This can lower peak memory when each value is consumed and then discarded. It does not guarantee low memory in every program: values retained by the caller, internal buffers, or converting the generator to a list can still use substantial memory. Generators also add little value for tiny fixed collections that are simpler as lists.

Read a large file one line at a time

readlines() loads all lines into a list. A file object is already iterable, so a loop processes lines without reading the whole file into memory:

with open("app.log", encoding="utf-8") as log_file:
    for line in log_file:
        process(line)

You can also write a generator that transforms or filters each line as it is read:

def error_lines(path):
    with open(path, encoding="utf-8") as log_file:
        for line in log_file:
            if "ERROR" in line:
                yield line

for line in error_lines("app.log"):
    process(line)

The file stays open while the generator is being iterated and closes when iteration completes. If a consumer may stop early, close the generator explicitly so the file is released promptly:

from contextlib import closing

with closing(error_lines("app.log")) as lines:
    for line in lines:
        process(line)
        if should_stop(line):
            break

Generators with loops and streams

Generators work naturally with loops that calculate each value on demand:

def even_numbers(limit):
    for number in range(limit):
        if number % 2 == 0:
            yield number

for number in even_numbers(10):
    print(number)
# 0, 2, 4, 6, 8

A generator can also describe an unbounded sequence. Consume it with a stopping condition; an infinite generator will not finish if passed directly to operations that wait for all values, such as list().

def counter():
    number = 1
    while True:
        yield number
        number += 1

for number in counter():
    if number > 3:
        break
    print(number)

Common uses

• Large datasets: transform rows one at a time in data pipelines instead of collecting every intermediate result.
• File and log processing: filter or parse records as the file is read.
• API pagination: fetch and expose one page of results at a time when the API and consumer support incremental processing.
• Streaming data: produce events or sensor readings as they arrive, with an explicit way to stop or handle failures.
• Lazy transformations: chain iterators so values are produced only as the final consumer requests them.

Common mistakes and best practices

• Expecting a list: a generator supports iteration, not list indexing. Convert it with list(generator) only when the full result is small enough to fit comfortably in memory and you need to reuse or index it.
• Reusing an exhausted generator: generators are usually one-shot iterators. Once consumed, iterate over a new generator created by calling the generator function again.
• Materializing values too soon: wrapping a generator in list() removes the streaming memory advantage.
• Using a generator for tiny fixed data: a straightforward list or tuple is often clearer when all values are small and immediately available.
• Ignoring cleanup: if a generator owns a file or another resource and iteration may stop early, close it or manage the resource at the caller.
• Forgetting that work is lazy: exceptions and side effects inside the generator occur during iteration, not when the generator object is created.