Python Generators use cases

Hi Friends recently I have been studying generators in python. During my course of study I found few really cool use cases for use of generators. In this post I will be describing few use cases where generators could be useful.

Eager Parsing of List

Suppose we have a function which does some computation and returns a set of data. In real world scenario we could relate it to a function which does some database operation and returns a list of entry from database. Below is the sample code.

# Scenario 1 without generators

from time import sleep
def load_data:
    data = list()
    for i in range(10):
        data.append(i)
        sleep(0.5)
    return data

data = load_data()
for _ in data:
    if _ > 5:
        break
    print _        

Above code calls load_data function, the function above iterates 10 times and in each iteration it sleeps for 0.5 seconds. So total time taken by function is 5 seconds. Now if see our code after the call to load_data we are looping through the list and once the value of list is greater than 5 we break, which means we don't need the value of list which is greater than 5. If we had a mechanism which could generate the value of the list one by one and return to the lower for loop. We could have saved 2.5 seconds of our runtime. This is where our generator comes to the party to help us. Lets change the above code with help of generator.

# Scenario 2 with generators

from time import sleep
def load_data:
    for i in range(10):
        yield i
        data = load_data()
    
for _ in load_data():
    if _ > 5:

        break
    print _

Now with the help of generators when the lower for loop is executed it calls the load_data function which returns the value of 'i' one by one. Example when the yield is executed it return '0' and control goes back to the lower for loop and it is executed and 0 get printed on the screen. Now when one iteration of lower for loop is completed control again to the upper for loop now it returns '1' and control goes back to lower for loop and 1  get printed on the screen. Similary loop will continue until i is greater than 5.

Advantage of Generators:

• In Scenario 1 we see that even we don't need all the list data is getting generated eagerly, so we can use generators for lazy list generation in Scenario 2.
• Another advantage of generator is on the memory used by our code. As we can see in Scenario 1 we are using a list to store all our integers so even if integer were to take 1 byte first code would need 10 bytes of memory, but second code will take memory to store only one integer. Suppose we were to run load_data loop for number as large as 100000000, in that case our program will need more memory if generators were not used.
• What we see from genertor runtime behaviour is that we call generator it does its task and return control back to us. Again we call generators and it performs its task and return control back to us.  So generators can also be used for interleaving of task between two different end points.

I will add few more use case of generators in next post. Thanks for passing by :)