Generator use-cases

Neil Mix nmix at pandora.com
Sun Mar 30 20:20:57 PDT 2008


On Mar 30, 2008, at 12:58 PM, Jason Orendorff wrote:
> Here's a brief case:  generators let you factor a complex loop,
> dividing the value-producing part from the value-consuming part.
> Neither part has to be transformed in a non-obvious way.  With
> iterators alone, you do this by rewriting the value-producing part as
> an Iterator class.  This obfuscates the iterator implementation.

Here's a concrete example of this: fetching and aggregating rows from  
a database.  You want to fetch rows from large queries in bulk (say  
10000 at a time) for perf reasons, but you want to hide that from your  
consumer which wants to see things one row at a time.

I recently wrote some python code that aggregated and integrated two  
data sources where performance was critical.  I had two SQL queries  
ordered by id (multiple rows per id).  Needed to aggregate each query  
by id and merge the results when the same id was in both results.   
Datasets were way to large to fit in-mem, and it was too slow to let  
the DB do the work spooling to disk.  So I created a generator for  
each query that fetched forward in bulk and aggregated by id, and a  
third generator that pulled records from the two query generators and  
merged as necessary.  The top-level code was then able to iterate  
simply one record at a time by pulling from a generator, oblivious to  
all the buffering, aggregating, and merging going on beneath.  Elegant  
and easy.  Had I not had generators at my disposal I might still be  
writing that script.  ;)

Krys mentioned the "advanced" (some would call it "abusive") mock- 
coroutining on my blog.  Setting that aside as perhaps an over- 
indulgence in generator goodness, I'm a big fan of generators for very  
practical use-cases like the one I mentioned above.  Especially for a  
language that lacks blocking IO facilities in most implementations,  
plus no concurrency, I'm certain generators would find a host of real- 
world practical uses.  It would be a shame if they didn't make it into  
ES4.



More information about the Es4-discuss mailing list