nmix at pandora.com
Sun Mar 30 20:20:57 PDT 2008
On Mar 30, 2008, at 12:58 PM, Jason Orendorff wrote:
> Here's a brief case: generators let you factor a complex loop,
> dividing the value-producing part from the value-consuming part.
> Neither part has to be transformed in a non-obvious way. With
> iterators alone, you do this by rewriting the value-producing part as
> an Iterator class. This obfuscates the iterator implementation.
Here's a concrete example of this: fetching and aggregating rows from
a database. You want to fetch rows from large queries in bulk (say
10000 at a time) for perf reasons, but you want to hide that from your
consumer which wants to see things one row at a time.
I recently wrote some python code that aggregated and integrated two
data sources where performance was critical. I had two SQL queries
ordered by id (multiple rows per id). Needed to aggregate each query
by id and merge the results when the same id was in both results.
Datasets were way to large to fit in-mem, and it was too slow to let
the DB do the work spooling to disk. So I created a generator for
each query that fetched forward in bulk and aggregated by id, and a
third generator that pulled records from the two query generators and
merged as necessary. The top-level code was then able to iterate
simply one record at a time by pulling from a generator, oblivious to
all the buffering, aggregating, and merging going on beneath. Elegant
and easy. Had I not had generators at my disposal I might still be
writing that script. ;)
Krys mentioned the "advanced" (some would call it "abusive") mock-
coroutining on my blog. Setting that aside as perhaps an over-
indulgence in generator goodness, I'm a big fan of generators for very
practical use-cases like the one I mentioned above. Especially for a
language that lacks blocking IO facilities in most implementations,
plus no concurrency, I'm certain generators would find a host of real-
world practical uses. It would be a shame if they didn't make it into
More information about the Es4-discuss