Generators

Generators

Generators are iterators, but you can only iterate over them once. It’s because they do not store all the values in memory, they generate the values on the fly. You use them by iterating over them, either with a 'for' loop or by passing them to any function or construct that iterates. Most of the time generators are implemented as functions. However, they do not return a value, they yield it.

3 Types of Generators - Pythonic Syntactic Sugar!

As I learned more about Python’s iterator protocol and the different ways to implement it in my own code, I realized that “syntactic sugar” was a recurring theme.

You see, class-based iterators and generator functions are two expressions of the same underlying design pattern.

Generator functions give you a shortcut for supporting the iterator protocol in your own code, and they avoid much of the verbosity of class-based iterators. With a little bit of specialized syntax, or syntactic sugar, they save you time and make your life as a developer easier:

Source from dbader

This is a recurring theme in Python and in other programming languages. As more developers use a design pattern in their programs, there’s a growing incentive for the language creators to provide abstractions and implementation shortcuts for it.

That’s how programming languages evolve over time—and as developers, we reap the benefits. We get to work with more and more powerful building blocks, which reduces busywork and lets us achieve more in less time.

Generator class

class BoundedRepeater:
    def __init__(self, max_repeats):
        self.max_repeats = max_repeats
        self.count = -1

    def __iter__(self):
        return self

    def __next__(self):
        if self.count >= self.max_repeats:
            raise StopIteration
        self.count += 1
        return self.count


generator_class = BoundedRepeater(4)
for x in generator_class:
    print(x)

If you’re thinking, “that’s quite a lot of code for such a simple iterator,” you’re absolutely right. Parts of this class seem rather formulaic, as if they would be written in exactly the same way from one class-based iterator to the next.

You’ll find that for most types of iterators, writing a generator function will be easier and more readable than defining a long-winded class-based iterator.

Generator function

• Generator functions are syntactic sugar for writing objects that support the iterator protocol. Generators abstract away much of the boilerplate code needed when writing class-based iterators.
• The yield statement allows you to temporarily suspend execution of a generator function and to pass back values from it.

def generator_function(max_repeats):
    for i in range(max_repeats):
        yield i

for item in generator_function(5):
    print(item)

It is not really useful in this case. Generators are best for calculating large sets of results (particularly calculations involving loops themselves) where you don't want to allocate the memory for all results at the same time. Many Standard Library functions that return lists in Python 2 have been modified to return generators in Python 3 because generators require fewer resources.

Generator Expressions

In Python, generators provide a convenient way to implement the iterator protocol. Generator is an iterable created using a function with a yield statement.

• The main feature of generator is evaluating the elements on demand. When you call a normal function with a return statement the function is terminated whenever it encounters a return statement.
• In a function with a yield statement the state of the function is “saved” from the last call and can be picked up the next time you call a generator function.
• Once a generator expression has been consumed, it can’t be restarted or reused. Pointer指去邊到就會停係個到。

Generator expression allows creating a generator on a fly without a yield keyword. However, it doesn’t share the whole power of generator created with a yield function. Generator expressions are best for implementing simple “ad hoc” iterators. For complex iterators, it’s better to write a generator function or a class-based iterator.

The syntax and concept is similar to list comprehensions:

gen_exp = (x ** 2 for x in range(10) if x % 2 == 0)
for x in gen_exp:
    print(x)

>>
0
4
16
36
64

Generator Expressions vs List Comprehensions

In terms of syntax, the only difference is that you use parentheses instead of square brackets. However, the type of data returned by list comprehensions and generator expressions differs. The main advantage of generator over a list is that it takes much less memory. We can check how much memory is taken by both types using sys.getsizeof() method.

We can see this difference from below example because while list creating Python reserves memory for the whole list and calculates it on the spot.

Generator calculates on the fly, so it is smaller.

In case of generator, we receive only ”algorithm”/ “instructions” how to calculate that Python stores. And each time we call for generator, it will only “generate” the next element of the sequence on demand according to “instructions”.

Generator calculates on the fly, so it is slower

On the other hand, generator will be slower, as every time the element of sequence is calculated and yielded, function context/state has to be saved to be picked up next time for generating next value. That “saving and loading function context/state” takes time.

from sys import getsizeof

list_comp = [x ** 2 for x in range(10) if x % 2 == 0]
gen_exp = (x ** 2 for x in range(10) if x % 2 == 0)

print(list_comp)
print(getsizeof(list_comp))
print("="*40)

print(gen_exp)
print(getsizeof(gen_exp))

print("-"*40)
print(next(gen_exp))
print(next(gen_exp))
print(next(gen_exp))

# Alternatively, you can also call the list() function on a generator expression
# to construct a list object holding all generated values:
print(list(gen_exp))

>>
[0, 4, 16, 36, 64]
128
========================================
<generator object <genexpr> at 0x7fec64539a98>
88
----------------------------------------
0
4
16
36
[64]