Python Generators

Before reading this article please refer my blog article on Iterators

  • Generators gives us values one at a time, instead of giving all values at once
  • Generators are iterables
  • We can use Generators only once, because once we request for the next value of generator, the old value is discarded and finally once we go through the entire generator, it will be discarded from memory also.
  • We can create Generators using Generator Functions and Generator Expressions
  • Generator Functions uses yield to produce value one at a time
  • Generator Expressions need parentheses (), which are similar to list comprehensions where we use square brackets []
  • We can use next() function and for loop to get values from generators
  • Generators are lazy because they give us a value only when we request it.
  • Generators are memory efficient and can be used for reading big data files

Generator Functions

# Regular Function uses return to return the value
>>> def fun_name():
...     return "Girish"
...
>>> fun_name()
'Girish'

>>> next(fun_name())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object is not an iterator

# Generator function uses yield to return the value one at a time
>>> def gen_name():
...     yield "Girish"
...
>>> gen_name()
<generator object gen_name at 0x0162E0B0>

>>> next(gen_name())
'Girish'

Assign the Generator Function Object to a variable (here in our below example it is myGenObj ) and continue passing the variable into next() and get the yield statements.

>>> def gen_name():
...     yield "Girish"
...     yield "Sanjay"
...     yield "Ramesh"
...     yield "Raghu"
...     yield "Ganesh"
...
#Generator Object
>>> myGenObj = gen_name()
>>> next(myGenObj)
'Girish'
>>> next(myGenObj)
'Sanjay'
>>> next(myGenObj)
'Ramesh'
>>> next(myGenObj)
'Raghu'
>>> next(myGenObj)
'Ganesh'
>>> next(myGenObj)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Generator Expressions

Before going into Generator Expressions, please refer my blog article on List Comprehensions

Generator expression is surrounded by parentheses () and the functionality is almost similar to List Comprehension which uses square brackets [].

Using Generator expressions we cannot perform list operations like indexing, slicing etc, as Generators are used to get output one at a time.

# Example of List Comprehension
>>> cube_lst = [n**3 for n in [2,4,6,8]]
>>> cube_lst
[8, 64, 216, 512]
>>> next(cube_lst)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'list' object is not an iterator

# Generator Expression
>>> cube_gen = (n**3 for n in [2,4,6,8])
>>> cube_gen
<generator object <genexpr> at 0x0163FA30>

>>> next(cube_gen)
8
>>> next(cube_gen)
64
>>> next(cube_gen)
216
>>> next(cube_gen)
512
>>> next(cube_gen)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

>>> cube_gen = (n**3 for n in [2,4,6,8])
>>> for x in cube_gen:
...     print(x)
...
8
64
216
512

We can use Generators to read files to give one line at a time. Like reading csv files.

I am using the below csv file (here testempdetails.csv ) read row by row using Generator Functions

S.No,Name,Designation,Experience
1,Ramesh,Manager,15
2,Sanjay,COO,20
3,Ananth,Senior Developer,8
4,Govind,Senior Testerf,8

Now let’s create Generator Function and read the csv file using open method.

>>> emp_data = "testempdetails.csv"
>>> rows = (row for row in open(emp_data, encoding ="ISO-8859-1"))
>>> next(rows)
'S.No,Name,Designation,Experience\n'
>>> next(rows)
'1,Ramesh,Manager,15\n'
>>> next(rows)
'2,Sanjay,COO,20\n'
>>> next(rows)
'3,Ananth,Senior Developer,8\n'
>>> next(rows)
'4,Govind,Senior Testerf,8'
>>> next(rows)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>> for row in rows:
...     print(row)
...
>>>
>>> rows = (row for row in open(emp_data, encoding ="ISO-8859-1"))
>>> for row in rows:
...     print(row)
...
S.No,Name,Designation,Experience

1,Ramesh,Manager,15

2,Sanjay,COO,20

3,Ananth,Senior Developer,8

4,Govind,Senior Testerf,8
>>>

References

  • https://docs.python.org/3/

Learn more about Python Advanced concepts in the next upcoming artilces.

Happy Learning!