Understanding the yield Keyword in Python

Understanding the yield Keyword in Python

The yield is a keyword that converts a function into a generator, which produces a sequence of values on demand instead of calculating all at once. The use of yield keywords makes a function generator which works with very less memory. For example if you have to read a big file line by line and process the data then you can use yield keyword. In this tutorial I will show you many examples of yield method.

When working with large datasets or complex streams or files it is not efficient to load all the data into memory. It is also impossible sometimes to load all data into memory due to the limitation of max memory we can have on a machine. In such scenarios yield keywords come into picture as a powerful feature that enables lazy evaluation. Due to the lazy evaluation feature, large amounts of data can be processed efficiently and with less memory. The lazy evaluation allows us to produce data on demand rather than computing all at once.

In this article we will deep dive into the yield keyword and understand how it can be used and how it differs from the return keyword. You will also learn how to use yield efficiently while developing code for large data processing.

1. What is the yield Keyword?

The yield keyword in Python which is used inside a function turning it into a generator, so that data is processed through lazy evaluation. A generator is a special type of iterator in Python that returns one value at a time. It pauses the function’s state between each call. 

When a function with yield keyword is called, it does not execute immediately. Instead, it returns a generator object, which can be iterated over using a for loop or the next() function. Here is the simple example of generator that prints 1 to nth numbers:

Example: A Simple Generator

def count_up_to(n):
    count = 1
    while count <= n:
        yield count
        count += 1

# Using the generator
for number in count_up_to(5):
    print(number)

Output:

1
2
3
4
5

Screenshot of the code:

Here, the count_up_to function yields numbers one by one. This program runs efficiently with less memory. Each time the function yields a value, its state (including variable values) is saved until the next iteration resumes. This way values are calculated lazily and perform well with less RAM usage.

2. How yield Differs from return

Now we will compare the return keyword with the yield keyword. The return keyword is used in a method to return all the calculated values. For example you are appending 100 items in a list and then returning the list, then all the list values are pre-calculated and returned from the function.

Example:

def use_return():
    return [1, 2, 3]

def use_yield():
    for i in range(1, 4):
        yield i

print(use_return())      # [1, 2, 3]
print(list(use_yield())) # [1, 2, 3]

Above are two functions that return the same output, but one is done using the return method while another function is developed using yield keyword. Outputs are the same, but the use_yield doesn’t build the entire list in memory at once, instead it generates values on demand. Python provides a powerful way of developing programs using the yield keyword that doesn’t build the entire dataset in one go.

Here is the output of the program:

3. Working Mechanism of yield

Now we will understand the working mechanism of the yield keyword. The working of yield is much different from return.

When a generator function is called:

1. It returns a generator object without running the function body.
   
   
2. When you iterate over it, the function executes up to the first yield.
   
   
3. The yielded value is returned to the caller.
   
   
4. On the next iteration, the function resumes right after the previous yield.
   
   
5. This continues until the function ends (raises StopIteration internally).
   
   

Example: Here is the Execution Flow

def demo():
    print("Start")
    yield 1
    print("Continue")
    yield 2
    print("End")

gen = demo()
print(next(gen))
print(next(gen))

Output:

Start

1

Continue

2

The generator pauses at each yield and resumes from where it left off.

4. Advantages of Using yield

Memory Efficiency

Generators yield one item at a time, avoiding storing large data structures in memory. Here is an example of a function that returns the generator which can be used to read files with large size.

def read_large_file(filename):
    with open(filename) as file:
        for line in file:
            yield line.strip()

This allows you to process files with millions of lines without memory overflow.

Infinite Sequences

yield makes it possible to create infinite sequences, which is not possible with lists.

def infinite_numbers():
    num = 1
    while True:
        yield num
        num += 1

for n in infinite_numbers():
    print(n)
    if n == 5:
        break

Pipeline Processing

Generators can also be chained together which helps in making complex data processing pipelines. Generators can be chained together to build efficient data pipelines. Here is an example:

def numbers():
    for i in range(10):
        yield i

def squares(nums):
    for n in nums:
        yield n * n

print(list(squares(numbers())))

5. Sending Values to Generators

It is also possible to send values to generators, which helps in developing rich applications in Python. Generators not only yield values out, but can also receive values from the caller using the .send() method.

def greeter():
    name = yield "What's your name?"
    yield f"Hello, {name}!"

g = greeter()
print(next(g))          # Start generator
print(g.send("Alice"))  # Send value to generator

Output:

What's your name?

Hello, Alice!

6. Using yield from

You can also call another generator with the yield from the statement. The yield from statement allows you to delegate part of your generator to another generator or iterable, simplifying nested loops.

Example:

def sub_generator():
    yield 1
    yield 2

def main_generator():
    yield from sub_generator()
    yield 3

print(list(main_generator()))

Output:

[1, 2, 3]

7. Real-World Use Cases

Here are a few use cases of yield in Python.

a) Reading Large Logs

You can use yield to read very large log files efficiently. Here is sample code:

def read_logs(filepath):
    with open(filepath) as f:
        for line in f:
            if "ERROR" in line:
                yield line.strip()

b) Streaming API Data

API calls can also be made through the use of yield from another generator which is calling the api. This way you can develop very efficient programs. Here is one simple example:

def fetch_data(api_pages):
    for page in api_pages:
        yield from call_api(page)

c) Producer-Consumer Pipelines

Generators can form chains where one generator feeds another, making them excellent for data streaming, machine learning pipelines, or ETL processes.

8. Key Takeaways

• yield turns a function into a generator.
  
  
• Generators are memory-efficient and stateful.
  
  
• They support lazy evaluation, producing values on demand.
  
  
• Use yield for large datasets, infinite sequences, or streaming data.
  
  
• Combine yield and yield from for cleaner, modular generator logic.
  
  

Conclusion

The yield keyword is one of Python’s most elegant features, providing a simple yet powerful way to handle data efficiently. Whether you’re processing files, streaming data, or building pipelines, using yield can make your code faster, cleaner, and more memory-friendly.